Skip to content

Developer Meeting Minutes

Jump to bottom
Bradley Dice edited this page Jan 10, 2024 · 65 revisions

Meeting Minutes

Typically collected by the rotating Community Manager. New information goes at the top.

2024-01-10

  • @bdice, @vyasr, @joaander, @cbkerr
  • Decided to add GitHub Discussions as a more permanent home for support discussions. All other Glotzer Lab packages have migrated there.
    • We will keep Slack alive, Discussions just provides another entry point that better aligns with how the Glotzer Lab is providing support for HOOMD, freud, etc.
    • Discussions will be added to signac, signac-flow, signac-dashboard, synced_collections.
    • https://github.com/glotzerlab/signac-docs/pull/198
  • Proposing @cbkerr for a maintainer role. Vote is open.
  • Brief discussion on some signac-flow refactoring on directives.

2023-05-17

  • @bdice, @b-butler, @cbkerr, @vyasr
  • We decided we want to remove triage manager role. @bdice will open the necessary PRs and remove it from the calendar
  • synced_collections should be ready to finalize. A couple of small changes outstanding
    • @b-butler brought up that the Changelog date needs updating
    • @bdice has a small fix for docs
    • @vyasr has conda-forge package ready
    • @vyasr has signac-docs PR ready
    • @bdice has PRs ready for signac to use synced_collections
  • @b-butler has some scripts to add to signac-flow to make it easier to test templates for submission in different environments.

2023-04-05

  • @bdice, @csadorf, @b-butler, @mikemhenry, @cbkerr
  • Work following up on signac 2.0 release: just signac-examples updates are needed, then we will continue as normal with PR reviews for open features.
  • Next time: Discuss a couple of roadmap documents from last year with @vyasr
  • signac-flow: We would like to finish up the open work, and then roadmap towards a 1.0 release
  • Submission script refactoring (https://github.com/glotzerlab/signac-flow/pull/722)
    • @csadorf: We've gone back and forth on whether logic should be in template (Jinja) or (environment) Python files.
    • Active signac developers using signac-flow submission (@b-butler, @cbkerr) should be willing to maintain all changes and functionality since no other maintainers have access to test submission functionality.
    • Changes should focus on maintainability and ease of use. Focus on the clusters and use cases we know users actually need.
  • Interest in signac development among current Glotzer lab members is limited. The current features seem to meet the needs of most users. (This is okay!)
    • Adoption is usually limited by friction - signac is just another tool to learn at the same time as simulation and analysis.
  • @cbkerr is working on a signac-dashboard PR using blueprints (https://github.com/glotzerlab/signac-dashboard/pull/182). @bdice will review when time allows.

2023-03-22

Meeting on zoom: @cbkerr, @b-butler, @mikemhenry, @bdice, @vyasr

signac 2.0 release

2023-02-08

Meeting on zoom: @csadorf (notes), @cbkerr, @b-butler, @mikemhenry

  • signac 2.0:
    • (Brandon) Blocked by verification that signac-flow still works; but it looks like it does. The migration path was pretty straight-forward.
  • Potential projects for new contributors:
    • Brandon:
      • signac sub-dataspaces
      • adding more flow hooks
    • Simon:
      • URIs
  • Is flow stable enough for 1.0?:
    • Corwin: Some users are discouraged by the amount and severity of breaking changes lately.
    • Discussion consensus: We should release flow 1.0 soon with a non-strict sem ver policy, i.e., breaking changes are defined as changes with major user impact.

2022-11-30

Meeting on Zoom, @cbkerr, @csadorf, @b-butler, @vyasr, @bdice

All efforts towards signac 2.0, then we'll shift to making signac-flow more maintainable.

We will update signac-flow and signac-dashboard together with just the changes to support signac 2.0 when it is released.

Release of signac-flow 0.23 will have support for signac 1.8 and address the FutureWarnings generated from signac 1.8. Deprecated features in signac-flow 0.23 will be removed in 0.24, and signac 2.0 support will be released in 0.25.

Devs feel free to chip on on PR reviews.

2022-11-02

Meeting on Zoom, @cbkerr, @b-butler, @vyasr, @mikemhenry

Some tasks still remain after merging next into master on signac. We added to that list with concrete items.

The biggest is reorganization of the internal package structure.

In addition, tasks remain in the following areas:

Docs:

  • Biggest changes are not giving projects names and changing root to path.
  • Need to make sure there is nothing misleading as people come back to docs following the upgrade

Flow:

  • Brandon needs people to review his PRs moving decorator logic to one place. These changes will make it easier to bring in new flow developers.

2022-10-05

Meeting on Zoom. @b-butler, @cbkerr, @vyasr, @bdice.

  • @b-butler finished 1.8.0 release process.
  • @vyasr will rebase next and merge next into master
  • Before 2.0 can be merged: https://github.com/glotzerlab/signac/milestone/2
    • A small PR is needed to remove deprecated config APIs
    • Package reorganization
    • Prepare signac-flow, signac-dashboard, signac-docs, signac-examples
  • Potential signac-flow API changes
    • Redesign of the package could be necessary in the future, due to rising technical debt
    • Long term, we might make aggregation part of the signac data model instead of part of flow.

2022-08-24

Meeting on Zoom. @bdice, @b-butler, @cbkerr, @kidrahahjo, @mikehenry Community manager: @b-butler

  • signac 2.0 and quicker release cycles
    • We discussed ramping up our release cycle by allowing smaller releases, even major ones.
    • Determined to release a 2.0 release of signac with current changes on the next branch sans changes that break flow. A tentative release scheduled for 2022-09-16.
    • Discussed a potential "hackathon" for a future 3+ release of signac that handles some of the larger changes that were planned for signac 2.0.
  • signac-flow API changes
    • Discussed @kidrahahjo's issue signac-flow #665 and came up with a modified proposal.

2022-07-27

Meeting on Zoom. @csadorf, @bdice, @b-butler, @cbkerr, @rskye144 Community manager: @mikehenry

  • FOMMS tutorial debrief (@cbkerr)

    • Presented poster on monday, directed people to workshop, ~1hr long
    • Went over pi calculation, motivated each step
    • tutorial covers aggregation
    • CLI tool example -> ffmpeg how to convert cat videos to cat gifs
    • mybinder ended up working pretty well
    • @mikehenry: Make issue to track mergiing it into other examples, then rewrite main tutorial with it

  • Signac 2.0

    • @bdice PR 731 do we still need it? @cbkerr Yes, need to change target to next
    • PR 423 Good first issue!
    • @bdice Need to test 2.0 with flow and dashboard
    • @cbkerr Close PR 642, make issue about it
    • @bdice We need to update pins for signac flow and dashboard for 2.0
    • @bdice We need to make sure examples work and show best practices
    • We won't back port fixes/make a compatibly layer, and will make flow major release with the right pin
    • Make sure we document version compatibility
    • Lets make a flow 1.0 release for signac 2.0
    • @cbkerr: will work to unify language around bundle, group, aggregation

2022-07-13

Meeting on Zoom. @csadorf, @b-butler, @vyasr, @cbkerr, @kidrahahjo, Community manager: @mikehenry

  • Final steps for 2.0 release, the 1.8 release.
    • @vyasr remove config API
    • @csadorf reorganize package (do last)
    • @kidrahahjo make synced collections an hidden import
    • We won't make a 1.x branch to backport things
  • @kidrahahjo what is the status of the tests in signac-flow?
    • @b-butler added a test for the template filter, feel free to do more work to increase coverage

2022-06-29

Meeting on Zoom. @bdice, @atravitz, @csadorf, @mikehenry

2022-06-15

Meeting on Zoom. @b-butler, @bdice, @vyasr

  • @b-butler showed demo of debugging in a container: https://github.com/glotzerlab/signac-docs/pull/167
  • One possible solution is to differentiate FlowGroup "options" into submit flags and run flags?
  • Refactoring flow config: https://github.com/glotzerlab/signac-flow/pull/649/
    • Planning to deprecate and remove config options for environment_modules and import_packaged_environments.
    • This will allow us to always get config information from the project. No config information will be needed outside the project.
    • Users will need to import their own environment modules, rather than having signac-flow automatically import them.
    • import_packaged_environments will always be enabled.

2022-05-31

Meeting on Zoom. @cbkerr, @b-butler, @bdice

  • signac-dashboard 0.3.0 is ready to release when someone (@cbkerr / @bdice) has time.
  • @cbkerr showed a draft of a poster he will be presenting at a soft matter physics summer school at UMass Amherst. @cbkerr will also be presenting at FOMMS.
  • @b-butler wrote a issue about how configs are used for signac-flow: https://github.com/glotzerlab/signac-flow/issues/643

2022-05-03

Meeting on Zoom. @cbkerr, @b-butler, @bdice, Community manager: @vyasr

  • @cbkerr will be presenting signac at FOMMS. He is also showing a poster at the soft matter summer school at Amherst.
  • signac 2.0 discussions:
    • We may want to get rid of all config-related APIs and only leave the CLI? We need to determine whether this is feasible for signac-flow (which could still interact with a config directly via configobj or another library).@b-butler is going to investigate.
    • We are going to remove synced collections without deprecation (there's no proposed replacement, it's getting spun out into a separate package).
    • We also want to reorganize the package into a flatter namespace once we've completed all other removals on next.
  • We need a signac 1.8 release to surface all the deprecations, especially because of the DeprecationWarning->FutureWarning conversion. There are two blockers right now:
    • Currently import signac throws FutureWarnings on master as a result of some DeprecationWarning->FutureWarning changes, but it's not clear exactly why. @bdice will investigate.
    • If we are going to remove all config APIs from signac we will need to deprecate those as well.

2022-04-19

Meeting on Zoom. @atravitz, @vyasr, @b-butler, @mikehenry, Community manager: @cbkerr

We discussed big picture topics within documentation.

  1. Ongoing work on glossary

    All are invited to comment on the draft glossary of signac terms..

    As we work toward signac 2.0, which will break things, Vyas mentioned that we might be able to change names if needed, and this is the time to do it. Brandon mentioned that we could implement new objects if the documentation effort implies that something is missing.

    Big picture questions include: when do we need to distinguish between the general concept, the python object, and the implementation (eg, how it appears on the file system or in a future MongoDB)

    Next steps: invite collaboration of the Google Doc glossary for feedback before scheduling a final review session to agree on terms.

  2. Linter for the documentation

    Goal: save reviewer time for common corrections and help check spelling and style.

    Corwin proposed Vale. It can be used with CI/pre-commit

    Corwin demoed possible checks, like for condescending terms ("simply") or enforcing certain terminology (like "state point" over "statepoint"). It's possible to ignore single instances.

    Next step: make with a small PR implementing and fixing the result of non-controversial checks as a base for future

2022-04-05

Meeting on Zoom. @vyasr, @b-butler, @csadorf, @bdice, Community manager: @cbkerr

  • signac at FOMMS
    • Corwin is planning to do a workshop to help convert peoples' scripts to signac.
      • Brandon advised to consider challenges of an audience of experienced MoSDeF users and new users. Corwin suggests enlisting experienced users to help with workshop.
      • Bradley: we could develop this workshop into a SciPy tutorial.
      • Bradley at FOMMS 2018
        • 25% of people familiar with signac ("MoSDeF crowd")
        • audience seems to be forward-thinking and open to trying software like signac
  • Corwin will prepare a release of signac-flow tonight.
  • GSoC
  • Breaking changes in 2.0 vs. leaving until 3.0
  • Future: We'll work on a migration guide as we update our own examples and based on change log entries.

2022-03-08

Meeting on Zoom. @atravitz, @vyasr, @b-butler, @cbkerr, @csadorf Community manager: @mikehenry

  • Note: If you get two approvals, you can merge your own PR
  • @cbkerr talk went well, replace Vyas scipy talk on web page?
    • @b-butler Make youtube playlist for all videos
    • @cbkerr will turn talk into content page on docs
    • @cbkerr talks about use cases
      • New users on HPC resources
      • Use signac to apply multiple models on the same dataset for ML
      • Migrate from unmaintained and unsupported workflow tools
  • @vyasr roadmap draft
    • signac (not flow)
    • ~6 month-ish roadmap
    • For 2.0 we should try and plan any breaking changes before 2.0 so post 2.0 we can just have minor releases
      • Address TODOs in code base: Need to identify them, scope them, turn them into issues, add comment linking them to issue number (@atravitz will tackle this)
      • Try and minimize breaking pain by scoping upcoming 2.0 changes
    • for 2.1 REST API
      • path is relative to host file path
      • need to figure out a good URI
      • https://github.com/glotzerlab/signac/pull/189
      • Working with different file layouts
      • Try and do as many non-breaking things we can , maybe create a prototype
      • Triage features that assume flat data structure
    • @atravitz & @cbkerr We do need to be careful to not add unnecessary complexity, users have a hard time already how flexible things are
    • @atravitz & @cbkerr We also need to make sure things are communicated in a way that doesn't make new users worried

2022-03-08

Meeting on Zoom. @atravitz, @vyasr, @b-butler, @cbkerr, @bdice Community manager: @mikehenry

  • @cbkerr Deprecation warnings popping up on tests, how should we handle this?
    • @vyasr use pytest's warning context manager to catch them, and make a PR adding a flag to error if the signac package throws any uncaught warnings.
  • An empty groupby doesn't throw an error, @cbkerr will make an issue
  • @cbkerr presenting at APS next week
  • We need to release hooks + docs
  • @vyasr
    • Schema migration going well
    • Check PR #865 to see if there are any APIs we can remove before 2.0
    • Rename synced collections module with an _ prefix and split into separate package namespace as part of 2.0 release
    • @vyasr will make a google doc to roadmap future signac, flow, and dashboard development
  • @cbkerr Should we do a beta release of 2.0?
    • Not worth the effort, we are slowly deprecating APIs and we will be ready for the bug reports that come in post 2.0 release
  • We are keeping PRs small so keep and eye out and keep reviewing them
  • Issue 283 Would be a good first issue once we pick a TOML parsing package
  • If you see any functions and think "why does this exist?" please make an issue! We want to remove as much of the API as we can before the 2.0 release

2022-02-22

Meeting on Zoom. @vyasr, @b-butler, @cbkerr, @klywang, @csadorf, @kidrahahjo, @mikehenry

  • @cbkerr's presentation to MoSDeF of signac-dashboard went well.
  • We discussed what to do with hook names, and the documentation which has not been merged yet.
    • Decided to change the names to match better with Python concepts and improve clarity.
    • Will go through a deprecation cycle.
    • We want names to consistently use nouns to denote the time when they are executed, e.g. on_failure rather than on_fail.
  • We agreed to remove checklists that act like radio buttons, and add a GitHub action that fails if a checkbox is not checked.
  • We discussed moving our internals away from DeprecationWarning in favor of FutureWarning since the former is targeted at developers and is hidden by default. We need to patch the deprecation package to support this change or vendor our own fork of it.

2022-02-08

Meeting on Zoom. Present: @bdice, @vyasr, @b-butler, @cbkerr, @csadorf, @kidrahahjo. Community manager: @bdice.

  • @Charlottez112 and @cbkerr show signac-dashboard demo of Project-level modules
    • Planning a PR. The dashboard is early-stage development, so it's okay to have breaking changes.
  • Not planning to present at SciPy this year.
  • Google Summer of Code plans
    • Feb. 7 - Feb. 21: application period for organizations. @vyasr will reach out to NumFOCUS.
  • Other package named signac
    • No action recommended.
  • signac 2.0 work
    • Keep scope of changes small in each PR.
    • After #667, delete Collection in a single PR.
    • Migration with directories #654 is merged into master
    • Vyas is doing some schema changes and deprecations on master (#675, #644), and some API changes (#678, #677, #674), on next
    • Next steps: large PR #643 will probably be split into 5-6 smaller PRs

2022-01-25

Meeting on Zoom. Present: @bdice, @vysar, @atravitz, @b-butler, @cbkerr, @csadorf

  • To focus on quicker iteration, we should keep PRs modular and branch off other changes that develop during code review.
  • Plan to merge PRs for signac 2.0 close to given state to get moving towards release.
  • Need to release new versions of flow and signac to get bug fixes and other changes available to PyPI and conda.

2022-01-03

Meeting on Zoom. Present: @bdice, @b-butler, @cbkerr, @kidrahahjo

  • New Community Manager
    • Manager roles stopped at the new year.
    • Brandon Butler (@b-butler) will start this year as Community Manager.
  • New Release Manager
    • Hardik (@kidrahahjo) will lead next release.
    • Rights to push to release branch needs to be given to committers. @bdice said he would change rights.
  • Hooks
    • Documentation (docs #154) needs to be merged before we can release flow v0.18.
    • Implement hooks from (flow #189) in new PR using the actually merged hooks implementation.
      • Do not both mutate and return project in install_hooks method.
    • Add hook usage to signac-examples. Wait until specific hooks PR is created and merged.
  • Status
    • Group status PR (flow #593) is ready. @bdice will look at benchmarks to ensure no performance regression.
    • We need to test status with differences in eligibility and submission status. @b-butler will create an issue.

2021-11-08

Meeting on Zoom. Present: @atravitz, @bdice, @mikehenry, @cbkerr, @vyasr

  • @bdice is working on benchmarking PR, will add documentation for running benchmarking in the Support and Development section

  • @vyas schema migration for v2.0

    • (Removing project id proposing providing passing project id as an optional argument as way to gently deprecate.
    • discussing difficulty of migrating to new versions that include parameters not defined in prior versions (already addressed by how the constructor is defined)

  • @cbkerr: addressed some performance questions regarding ~1000 jobs

2021-10-25

Meeting on Zoom. Present: @cbkerr, @vyasr, @bdice, @atravitz, @b-butler, @mikemhenry, @bc118, @ramanishsingh

  • @cbkerr is presenting signac at APS.
  • @b-butler has made a number of PRs to roll out Python 3.10 support.
  • Helped @bc118 diagnose a late binding issue in dynamic operation generation.
  • Worked with @ramanishsingh to narrow down potential cluster scheduler issues causing nearly trivial operations to slow down.

2021-10-11

Meeting on Zoom. Present: @cbkerr, @vyasr, @bdice, @atravitz, @b-butler

  • We'd like to add support for Python 3.10
    • No need to actively drop Python 3.6 support
    • CircleCI has a Python 3.10 image ready, we should be able to get that set up
    • We're fine keeping additional support beyond NEP 29, but we should consistently document that only NEP 29 is officially supported
    • We want to make sure we are always testing all supported versions, so we need to maintain Python 3.7 testing even if it requires dependency version pinning after NEP 29 drops 3.7
  • We're going to move towards using asv for signac benchmarking. @bdice put together a nice demo.
  • signac 2.0 is progressing. Most non-config work is nearly done.

2021-09-27

Meeting on Zoom. Present: @atravitz, @b-butler, @cbkerr, @vyasr, @bdice, @mikemhenry.

  • PR discussions/triage
    • flow pr #504 should we perform config file/schema validation? How can we validate a flow schema config without signac knowing about flow cofig options?
    • flow pr #562 remove pprofile since it is no longer supported & remove profile option
    • flow pr #548 closing in favor of desgine outlined in flow issue 547
    • @atravitz will review hooks doc

2021-09-13

Meeting on Zoom. Present: @b-butler, @cbkerr, @vyasr, @bdice, @mikemhenry.

  • @cbkerr is now on the website 🎉
  • Removing office hours
    • Announce office hours deprecation in general channel
    • Will make a weekly thread in support channel with a bot to invite people to schedule office hours
  • signac-flow and signac config
    • FlowProject config should be cached and immutable upon instantiation
    • Environment will read config from disk when instantiated, not when projected is instantiated.
    • Signac config cli api will control configuration for flow

2021-08-30

Meeting on Zoom.

Present: @b-butler, @cbkerr, @vyasr, @csadorf, @kidrahahjo

  • Discussed triage manager function. Reaffirmed focus on delegating existing tasks on existing PR's and issues rather than tackling all such tasks individually.
  • Presented CodeTriage as a tool for encouraging contributions. No decision was made on using it at the time.
  • Schema migration is on hold until @vyasr has the time to tackle it (#506, #527).
  • Discussed (#270) and plans to add intra-group parallelization.

2021-08-16

Meeting on Zoom. Present: @atravitz, @b-butler, @bdice, @cbkerr, Alex Lee.

  • Discussed updates for NumFOCUS newsletter.
  • Reviewed open PRs for signac-flow and planned 0.16.0 release.
    • print_status tests (#562) are in progress. The core tests are working and the PR just needs some cleanup.
    • Drexel University Picotte template (#561) is ready to merge. One more review would be good.
    • Groups in status check (#548) is paused until @bdice has time. Issue #547 shows the proposed design.
    • Raising errors for groups applied to non-operation functions (#544) needs a little bit more work but can be finished quickly and merged for 0.16.0.
    • @klywang is continuing to work on execution hooks (#508).
    • Schema versioning (#504) is waiting on @vyasr.
    • Benchmarks / profiling (#399) is paused until @bdice has time.
    • Issue #559 discusses a problem where DeprecationWarnings are handled as errors. This needs a minimal reproduction when @bdice has time to make one (the issue describes steps to test).
    • Issue #543 (returning scheduler job ids from FlowProject.submit) was removed from milestone 0.16.0. It is a non-blocking feature request with clear steps for how to proceed in the issue, and should not block a release.
  • In signac-flow, the cores_per_node class variable is defined by several environements but is not used except in Summit.
    • Should this information be in a template (Jinja) or an environment (Python)?
    • Could we create a block for defining these kinds of variables in a template that would simplify extending the template logic?
  • Updates on signac (2.0)
    • Planning to remove Collection.main, which seems unnecessary. As a note, the Collection class is only used once in Project._find_job_ids and once in schema.py's _build_job_statepoint_index. The Collection instances are created and consumed within the scope of the function, so it may make sense to make Collection private (and perhaps to rename, to avoid confusion with the new SyncedCollection classes which are unrelated).
    • @bdice noted an edge case with Project.export_to / Project.import_from failing when a signac Project has only one job with an empty state point {}. The automatic path detection returns an empty path, which puts the exported data at the top level of the exported archive. However, this doesn't work for the import function, which requires imported data to be in a directory. (Is this a bug in exporting or importing?)
  • @cbkerr will add a committer bio to the website this week.

2021-08-02

Meeting on Zoom. Present: @atravitz, @b-butler, @bdice, @cbkerr, @vyasr.

  • Discussed some open work on signac 2.0.
    • @bdice asked about issues #588 and #590.
    • On #588, the discussion summary can be found in this comment.
    • On #590, the discussion summary can be found in this comment.
  • Briefly discussed Expanse template #537, which is ready to merge.
  • @cbkerr (Corwin Kerr) is our newest committer! @bdice will walk through the onboarding process with him this week.

2021-07-19

Meeting on Zoom.

Present: @atravitz, @b-butler, @bdice, @cbkerr, @mikemhenry

  • Several of the committers/maintainers are in transitional periods. Thinking about long term maintenance of the package. The pace of new features is slowing.
  • signac 2.0
    • @atravitz is going to look at removing deprecated code
    • @bdice will help refactor core/contrib/common
    • Need to identify a point person (@vyasr?) to guide the changes related to synced collections in 2.0
  • signac-flow
    • Currently moving at a slow, steady pace. Needs continued investment for cluster templates and improved documentation.
    • Major ongoing work: Hooks
    • Keep developing the concrete mental model (docs, glossary)
  • @vyasr will update the rotating triage/community managers on the calendar

2021-06-21

Meeting on Zoom.

Present: @atravitz, @b-butler, @bdice, @csadorf (note taker), @kidrahahjo, @mikemhenry

  • Questions about PR 544

    • Bradley: Do operation functions need a name?
    • Brandon: It is always possible to explicitly assing a name as part of the @operation decorator.
    • Bradley: Using a lambda function, then defaults to operation name "lambda".
    • It will be necessary to determine whether it is possible to detect lambdas.
    • Bradley voices some concerns that we should not do much more function introspection.
    • It will be tried to raise a ValueError when a lambda type function is provided, but no operation name.

  • RuntimeError vs ValueError on FlowProject definition.

    • Bradley recommends to convert the two instances of RuntimeError to ValueError to achieve consistency.
    • It is proposed to introduce a custom error (e.g. FlowProjectDefinitionError) to make it even more explicit what is causing the issue.
    • General agreement on the above points.

  • PRs blocking the next (flow) release:

  • Release management nodes

    • Mike wrote up some notes on how to push releases for signac and signac-flow (https://docs.google.com/document/d/1qHUMQeFjG2GTSp5JHMHbVntFZPBpOuXFTRQT6nJi3tU/edit?usp=sharing)
    • The idea is to always have a team of two, where one person knows how to do releases and teaches the other.
    • Question about whether the responsiblity should be split by sub-project or not.
    • The technical steps for releasing are basically identical between the sub-projects.
    • However it might be difficult to convince people to take care of all sub-projects.
    • According to Bradley and Brandon, actually making a release is not technically challenging at the moment.
    • For now there will be only one release manager (plus assistant) for all sub-projects.

2021-06-07

Present: @bdice (notetaker), @atravitz, @klywang, @b-butler, @mikemhenry

  • Ready to release signac 1.7.0. (Release managers: @bdice and @b-butler)
  • signac-flow is mostly ready for 0.15 aside from aggregation.
  • Remaining aggregation tasks: refactor AggregatesCursor (@bdice), docs, examples.
  • Improve docs for templates (suggested by @atravitz). A tutorial or working code example would be better than the current text-heavy page. Perhaps show filesystem tree and file names for the existing code snippets.
  • @mikemhenry will prepare a guide for release manager duties.
  • @bdice explained motivation for signac-flow PR #533, all seemed to be in favor of the design change.

2021-05

Developer meetings this month were focused on content for the 2021 SciPy proceedings paper.

2021-04-26

Meeting on Zoom.

Present: @vyasr (notetaker), @mikemhenry, @b-butler, @bdice, @atravitz, @kidrahahjo

Signac Meeting Notes

  • Release 0.14:
    • is ready. @b-butler is the Assistant Release Manager this time around, so he's going to pair the release with @bdice
    • The Expanse template is probably going to get skipped for this release. We'll target that at the next version.
  • operation directives
    • We are considering deprecating the directives decorator in favor of operation.with_directives in the long run to make groups and operations more alike. However, we want a long deprecation period for that, and will probably target this at 1.0 and not sooner.
    • Currently @directives accepts keyword arguments, whereas group.directives expects a dictionary. This inconsistency will become more obvious when operation.with_directives becomes a standard. Since operation.with_directives will require a dictionary like group.with_directives, we should verify our API choice here. The decision for groups was made very intentionally, but we've forgotten the reasons and want to remind ourselves.
  • We need to be more active about involving reviewers constructively. Committers need to be cognizant not to overload themselves by reviewing too many things they are assigned to. Gentle nudges of requested reviewers who are taking a while are encouraged (emphasis on gentle! We want to encourage them, not scare them off)
  • Schema changes
    • Everyone is in favor of removing signac project name
    • Everyone is in favor of modifying the submission id for cluster jobs to use FlowProject class names in place of Signac project names. When we do this rename, we should take the opportunity to re-engineer the cluster job names to be as readable as possible. One suggestion was to put operation name closer to the beginning of the submission name for clarity.
    • People seem generally in favor of storing all file data associated with a FlowProject in the signac project directory, rather than wherever project.py is.
    • project.py probably can possibly still be located elsewhere, but we'll need to revisit that discussion later to figure out what caveats there are
    • The full flow schema discussion is now documented on #506

2021-04-12

Meeting on Google Meet.

Present: @vyasr (notetaker), @mikemhenry, @b-butler, @bdice, @atravitz, @kidrahahjo

  • Plans for Next Release (0.14)
    • Focus of 0.14 will be fixing regressions in 0.13 (e.g. the breaking of add_operation and parallel status checks).
    • Some minor features new features (memory directive, submission summary) and cleanup tasks (improving templates and environments).
    • @bdice is working on rewriting Jinja templates and will finish that up.
    • @kidrahahjo offered to help out with the --only-incomplete-operations bug, needs to touch base with @bdice on that.
    • We need to add regression tests for status output, probably in the style of our testing of submission scripts. We'll target this for 0.15 (see #435).
    • @vyasr is planning to work on schema migration. @atravitz offered to help out or to find help if there are good intermediate steps.
    • We'll try to squeeze in Expanse template (@cbkerr is working on it), but it's not a must and can be pushed to 0.15.
    • Main upcoming feature is aggregation. We'll have a follow-up meeting next Monday at the same time to discuss next steps. Aggregation will be targeted at 0.15.
  • Meetings
    • Still planning to move to Zoom, @bdice is taking care of that for the moment.
    • @mikemhenry has an idea of how Zoom meetings can be set up so that people outside the organization can host the meeting. That would be worthwhile to set up to spread the load on meeting hosting.
    • The longer-term plan for meetings (and Slack) is NumFOCUS sponsorship, but we don't have a timeline for that right now.
  • SciPy Paper - @b-butler, @bdice
    • Interested committers: @b-butler, @bdice, @atravitz, @kidrahahjo, @csadorf, @vyasr, and @mikemhenry.
    • Possible topics: groups, synced collections, aggregations, new governance, new templates/environments,
  • CSV Talk - @atravitz
    • @atravitz is planning to present a higher-level view.
    • @vyasr suggested pulling some of the early overview slides from earlier presentations, in particular the animated slide showing a slowly changing data space and workflow and how difficult it can be to adapt file naming without signac.
    • @atravitz wants to use SciPy paper figures for signac-flow since we have fewer flow-specific figures, which should help.

2021-03-15

Meeting on Google Meet.

Present: @mikemhenry (notetaker), @klywang, @Charlottez112, @cbkerr, @b-butler, @bdice, @vyasr

  • Tabled for next meeting:
  • Transitioning the project media/abstracts/presentations to either Google Drive or Dropbox (University of Michigan is not renewing their contract with Box)
    • No strong objections to using Google Drive (personal accounts not enterprise) to store materials in a shared folder.
      • @bdice will take the lead on this migration
      • We will document how to use the drive on the wiki
  • New Role: Release Manager
    • @bdice is the de facto release manager for the signac project. We need to make this a rotating role (like triage and community manager) so that others are trained in cutting releases and he can take a well deserved break from this responsibility.
    • The role will not be in charge of deciding when to make a release or what goes in the release, that will be determined at our developer meetings with input from all members of the project.
    • @mikemhenry will add a draft of this role to the governance documentation
    • We will also create an assistant release manager who will work with the release manager to be trained and take over their role after N releases. @b-butler will be the first assistant release manager.
    • We will create a new PR template for release PRs that will include the checklist for creating a new release.
    • At least the release manager or assistant release manager will need to be a maintainer to have access to some of the processes involved (for example pypi access)
  • The 0.13 release of flow has some issues. We will focus on fixing these issues/merging PRs for the 0.14 release before working on new features

2021-03-15

Meeting on Google Meet.

Present: @cbkerr (notetaker), @b-butler, @bdice, @atravitz, @csadorf, @kidrahahjo, @vyasr

  • signac 2.0

    • Vyas: signac 2.0 requires resolving open issues on schema. See issue #527 that includes decoupling signac from flow.
    • Bradley wants to do more deprecations on signac API soon.

  • Bradley and Josh fixed threading issue last week. See issue #528, PR #529, PR #530.

    • SyncedCollection is meant to be thread safe, but locking needed to be lifted into Job class to accommodate lazy loading. The problem could occur when opening jobs by id when no state point is in the project cache.
    • Simon asked if forking was needed to trigger? Answer: Problem could be reproduced without forking.
    • Simon asked if the backend (JSONCollection) is truly thread safe if it requires management by the owner of the JSONAttrDict instance?
    • Vyas beginning to describe, has connection issues. Each instance needs a reference to the same lock.
    • Bradley's description: Collection instance locks prevent inconsistent data state between modifications. Locks at local job class prevent creation of multiple collection instances.
    • Simon: Are locks associated with paths on file system? Bradley: Yes, but we don't use realpaths (with symlinks resolved) because it is expensive.
    • Simon: Does the lock need a unique path? Will see the PR for this.

  • Work on execution hooks continues with Corwin and Kelly.

  • Bradley planning signac-flow release 0.13 this week for Bridges-2 support for containers. Walltime PR #476 should be completed first. (Thanks Hardik!)

  • Hardik - figuring out how parallelization works within/between groups.

  • Hardik asks if we want labels for aggregation.

    • Brandon agrees this makes sense.

    • Bradley: We can follow the implementation of aggregators/groups for aggregators/label functions.

      Definitions for signac-flow:

      • Bundling: submit multiple operations in one script (parallel or serial).
      • Group: submit a set of operations even if dependencies are not met (evaluate dependencies at runtime, re-evaluate as operations are completed).
      • Aggregation: run an operation on multiple jobs in the FlowProject. Used for analysis (The basic idea is for job in jobs:). The default aggregator creates aggregates of one job. Vyas suggests looking at issue #274.

      Proposal from Bradley: Should label functions accept the aggregator as a decorator/argument? This would allow label functions to be treated like operations.

  • Alyssa is working on glossary to go along with concepts page. We don't want people to have to search through multiple pages to figure out things. The process of writing a glossary is helpful for developers too. Take a look at this Google Doc if you have a chance.

    • Key confusion from office hours: heterogeneous is good but not incoherent –> More heterogenous parameter space examples need to be highlighted because users might (reasonably) assume that signac requires grid-based schemas with homogeneous keys.

  • Simon anecdote: Used signac to organize benchmarking for some software compatibility checking.

  • Vyas requests for Alyssa (on triage) to prioritize triaging signac-flow issues.

  • We need to send out NumFOCUS updates this month. Mike (@mikemhenry): expect a follow up in Slack.

2021-03-01

Notes by @mikemhenry

  • Attendees: @bdice, @cbkerr, @atravitz, @b-butler, @csadorf, @mikemhenry
  • Second meeting on Google Meet. Everything went smooth except for @csadorf had some audio issues at first.
  • Folder and file hierarchy for signac https://github.com/glotzerlab/signac/issues/197
    • Things that don't belong in source control will be in .signac/ (shell history, state point cache, bundles from signac-flow)
    • Project config signac.rc stays in root of project.
  • Discussion on signac_statepoints.json:
    • Deprecate Project.write_statepoints, Project.read_statepoints, and index arguments throughout signac code base
    • Repairs will be performed using .signac_sp_cache.json.gz, we should add recommendations to run signac update-cache or Project.update_cache() after initializing jobs.
  • Information shown by status
  • dependabot protocol
    • Approve PR
    • Turn on auto merge (squash)
    • Wait for the bot to rebase and re-run CI

2021-02-15

Notes by @bdice

  • Attendees: @bdice, @vyasr, @cbkerr, @atravitz, @tcmoore3, @kidrahahjo, @b-butler
  • First meeting on Google Meet. Everything was smooth.
  • We will move office hours 1 hour earlier (noon EST) to avoid conflicts for @cbkerr and others.
  • @atravitz leads discussion on docs:
    • During office hours, we discussed creating a revised Concepts page
    • Progressive steps of building the concept
    • Keep audience in mind. Who would read the concept page?
    • The Concepts page should convey aim/purpose, establish mental model, and introduce core terminology
    • We already have many definitions in the topic guides but need more inter-linking
    • Need to invite @klywang to future meetings, since she has worked on diagrams
    • We are seeking input on and revisions of tutorial content from anyone who is interested!
  • @kidrahahjo asked about default directives for memory, walltime:
    • We want to move towards using directives and deprecating CLI arguments
    • Walltime directive support is probably incomplete
    • Improves self-documenting behavior of the code
    • Need to clearly document usage patterns for all directives, but especially memory/walltime

2021-02-01

Notes by @bdice

  • Planning to Google Meet for conferencing
  • @vyasr summarized synced collections
  • Discussed what else to do for signac 2.0
    • Integrated query (#332)
    • Re-organize internal files (signac.rc, etc. #197)
    • Subpackage re-organization (everything in top level instead of contrib, core, common)
  • signac: We'll merge synced collections (#484) into master and use that as a testing ground
  • Discussed proposed signac-flow status output for aggregates from @kidrahahjo. @vyasr and @bdice and @mikemhenry are in support, in favor of combining tables.
  • signac-flow: refactoring _AggregatesCursor to use all aggregates.

2021-01-18

Notes by @b-butler (with help from @cbkerr)

  • Discussed plan for next office hour (CI practices and testing our oldest supported dependencies).
  • We need to push on more extensive documentation with examples and tutorials without worrying to much about being prescriptive. We agreed to focus on this after finishing the two GSoC projects. An example is documenting Project.to_dataframe and having a tutorial on signac's integration with pandas.
  • Planned on releasing signac-flow v0.12 without user-facing aggregation but with the internals changed to allow for aggregation.

2021-01-04

Notes by @b-butler

  • Discussed year end signac tweet.
    • Mentioned that we may want to store the information on how to get metrics for said tweet for future years.
  • Discussed why workspaces should be a subdirectory of the signac-flow project directory, and that some users would like to still be able to name the workspace directory.
  • Talked about how to address status checking performance in signac-flow next.
    • Implement a bidirectional mapping for groups and their aggregates signac-flow PR #415.
    • Create a method for efficiently iterating over group, aggregate pairs.
    • A general refactoring the status checking methods to include the first two points.
  • Creating more performance benchmarks for signac-flow to prevent drastic regressions in future PRs.

2020-12-21

Notes by @bdice

  • Discussed ongoing work on PRs.
  • signac-flow next will be merged into master once the performance is roughly equivalent. The status checking code is slow but running and submitting operations appears to be fine, from @bdice's tests.

2020-12-07

Notes by @bdice

  • Spent most of the meeting discussing ongoing work to refactor and improve signac-flow by Bradley and Hardik.
  • Discussed signac-flow #383.
  • Bradley asked some questions that came up while writing docs (from this PR, many of the answers are also provided in this comment)
    • Vyas and Brandon described the reasons for some current behaviors related to operations defined at the instance level and class level.
    • The @operation decorator defines operations at the class level, which are registered at the time of FlowProject instantiation. The add_operation method only adds instance-level methods. This code should probably be refactored a bit to make those designs clearer, and to prevent the possibility of creating conflicting names for operations/groups.
    • Immediate next steps: continue fixing up documentation, make a pass to deprecate public API that shouldn't be public.
    • In the next month: create benchmarks for the "core" API and compare performance / profile on all released versions since 0.8, master, and next.

2020-11-23

Notes by @atravitz

  • Brandon submitted talk to MRS, will put slides in the shared folder

  • Bradley did pre-commit upgrades: linked here

  • Aggregation and refactoring

    • PR with all jobs being "aggregates of one" is now mergedPR #335)
    • Next steps for code clean up: next 1-2 weeks
      • look for places to make improvements and create issues
      • Brandon mentioned removing single-use helper functions where they aren't necessary
      • Vyas suggested looking at removing parallelization for clarity, this can be done as-needed
      • a good starting place is the aggregation to-do list

  • Next meeting: discuss signac 2.0 release

2020-11-09

Notes by @atravitz

  • pyupgrade (PR #413)

    • this draft should be a PR, include this and possibly black as well

  • Brandon will present at 2 minute, pre-recorded poster presentation at MRS

    • make any edits/suggestions by tomorrow

  • Aggregation plans and timeline (PR #335):

    • merge aggregation into next branch when first pass is done (treating everything as aggregates of one)
    • do refactoring to improve performance, all on the next branch
    • finalize aggregation on next then merge

2020-10-26

Notes by @vramasub

  • We need to improve discoverability of docs. Ideas:
    • Include inheritance in things like JSONDict (created PR #398)
    • Add more links between different methods
    • Look into whether it's possible to get code examples (in the documentation) to link to the API docs
  • If the job doc is not intended to store large quantities of data, we should maybe mention that more frequently because it otherwise might seem like the obvious thing to do.
  • Add a brief outline of the entire tutorial at the beginning to show what features exist and how they ought to be used.
  • Sometimes workflows inherit from FlowProject, while others directly add operations to FlowProject. We should probably be more consistent here, and we might also want to discuss that in best practices. We also need to address places where the old API (add_operation) and the new API (decorator) are used interchangeably in examples.
  • The command line interface for flow is much easier to use than the Python interface. This is somewhat by design, and as we improve the Python API it should get easier to work with, but we may need to improve the documentation as well to show more examples of using it.
  • Hardik and Brandon will have a longer discussion about the design of the operations in Python and making them directly runnable (as opposed to passing them to FlowProject methods).

2020-10-12

Notes by @vramasub

  • Discussed release 0.11. All the appropriate deprecations made it in.
  • The parallel buffering bug in signac-flow leads to problems when large data spaces overflow the buffer during parallel status updates. We need to think about ways to employ some sort of locking mechanism to make this parallel-safe.
  • Improving parallel performance
    • Filtering is fast, but running flow itself is slow.
    • Could make -n calls faster by terminating early rather than doing the entire project's worth of work before filtering the number of jobs to use.
    • Should we document for users ways that it's super slow? At the moment it's not clear exactly what those are other than slow condition functions (which are already documented and possible to profile), maybe can add more in the future once we've optimized more.
  • Parallelization of status is not being used when run is called, so the run call is much slower right now. We should try and unify these more.
    • We need to figure out the right abstraction layers
    • We want to use the same status logic in run and submit, and ideally submit should take advantage of run as well.
  • We're going to try and finish up aggregation in the near term
    • Vyas is going to review the open PR first, then send over to Bradley and Brandon.

2020-09-28

Notes by @mikemhenry

  • hacktoberfest

    • See details here for maintainers (we are ready to participate)
    • Need to do a bit of research (@mikemhenry) to see what sorts of issues and pull requests we want to promote/label
    • Make a blog post or wiki page describing projects participation
    • Promote event on twitter

  • @kidrahahjo / Aggregation

    • Will have main aggregation PR split into two smaller PRs by the end of the week
    • Will make a push to the Directives PR (mostly doc changes) by the end of the day

  • 0.11.0 signac-flow release

    • Finalize what should be in the 0.11.0 milestone by 2020-10-02.
    • Make the 0.11.0 release by 2020-10-09

  • Longer Term signac and signac-flow goals

    • signac project board here
    • signac-flow project board here
    • Let's keep putting ideas on the project boards and then next meeting we can start refining.

2020-09-14

Notes by @mikemhenry

  • Helped Ramanish troubleshoot flow submission

    • Cluster had sbatch and qsubin path, so flow didn't know it was supposed to use sbatch
    • Ramanish submitted PR #353 to support the Minnesota Supercomputing Institute at UMN

  • The signac project has multiple "support" channels

    • Gitter, Our slack, and the MoSDeF slack
    • Hard to keep track of everything/eyes on everything
    • Gitter pro: lower barrier of entry than slack
    • No action taken

  • Google Summer of Code Wrap Up

    • @vishav1771 / Synced Collections:
      • Only one PR left #363
        • Followup discussion on slack + pull request
    • @kidrahahjo / Aggregation:
      • Error with tests on CI that didn't appear locally with with PR #335
        • Error now appears locally after cache was cleaned, working on root cause troubleshooting on slack

  • Roadmap for signac 2.0 and flow 1.0

    • signac
      • Remove everything that we have slowly been deprecating in 1.x
      • Very little API breakage, most disruptive will likely be the requirement that keys cannot be int or bool
      • synced collections
      • new backend options
      • schema version and schema migration
      • Let's put everything we think is needed for a 2.0 release in the project board here
    • flow
      • Not as clear as signac (do we need hooks? is the API stable enough?)
      • To work on this asynchronously we will put what we think should be in the 1.0 release here and in two weeks discuss

2020-08-31

Notes by @bdice.

  • Discussed Google Summer of Code projects with @kidrahahjo and @vishav1771:
    • @kidrahahjo / Aggregation:
      • PRs #335 and #336 have been converted to drafts until #348is completed.
      • Discussed equality comparisons, status checking.
    • @vishav1771 / Synced Collections:
      • Validation in PR #378 has a small amount of work to be done, will be ready for review soon.
      • @vyasr discussed a proposed model for buffering that might clarify both concepts and implementation.
  • Assigned community manager (monthly rotation) to @mikemhenry for September
  • Assigned triage manager to @mikemhenry for the upcoming week (through 2020-09-07) and @tcmoore3 for the following week (through 2020-09-14).

2020-08-17

Notes by @bdice.

  • Discussed Google Summer of Code projects with @kidrahahjo and @vishav1771:
    • @kidrahahjo's project: Aggregates are now defined on FlowProject initialization (instead of dynamically generated). We discussed CLI/Python API designs for specifying aggregates.
    • @vishav1771 will write standalone benchmarks for the new SyncedCollection classes and contribute them to signac-benchmarks.
  • Assigned community manager (monthly rotation) to @bdice for the remainder of August and triage manager to @b-butler for the upcoming week (through 2020-08-24).
    • Next up: @bdice becomes triage manager on 2020-08-24, @mikemhenry becomes community manager on 2020-09-01.
  • @atravitz and @cbkerr gave an update on documentation revisions. In progress.
Clone this wiki locally