Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add simple RNN functionality #48

Open
30 of 53 tasks
drhuffman12 opened this issue Dec 17, 2020 · 0 comments
Open
30 of 53 tasks

Add simple RNN functionality #48

drhuffman12 opened this issue Dec 17, 2020 · 0 comments
Assignees
Labels
RNN initially simple RNN

Comments

@drhuffman12
Copy link
Owner

drhuffman12 commented Dec 17, 2020

Add simple RNN functionality (See Issue: #15 )

  • Phase 1: initial prototype/skeleton (w/ basic structure calc's) (See PR: Drhuffman12/cmn basic rnn (part 1) #46 )
  • Phase 2: expand prototype/skeleton (to include the associated 'mini_net' objects) (See PR: Drhuffman12/cmn basic rnn (part 2) #47 )
  • Phase 3: expand prototype/skeleton (to include the methods to 'guess' and 'train') (See PR: Drhuffman12/cmn basic rnn (part 2) #47 )
  • Phase 4: FIX to/from json (use type key?) (in a later PR, at least partly in PR: Drhuffman12/cmn basic rnn (part 2) #47)
  • Add code to: (See PR: drhuffman12/cmn_basic_rnn_part_5 #51 )
    • Collect (split?) a 'sequence of input and output data' into small chunks for use by RnnSimple#train(..)
    • Add a variation of RnnSimple#train(..) that loops thru the 'sequence of input and output data'
  • add initial BreederUtils and Breeder (See PR: Drhuffman12/cmn basic rnn part 6 #54)
  • Consolidate error_distance related code into ErrorStats (See PR: drhuffman12/add_team_utils (part 1) #56 and https://github.com/crystal-lang/crystal/tree/master/.github/workflows)
  • Convert misc classes to use ErrorStats (See PR: drhuffman12/add_team_utils (part 1) #56)
  • Reorg/refactor props and inits (See PR: Drhuffman12/add team utils part 2 #57)
  • Add a net mixer
    • (?) ~~BreedParent
    • Breeder class(es) (renamed to *Manager classes)
    • (?) TeamUtils (See PR: Drhuffman12/add team utils part 4 #59)
      • Should mix misc combo's of nets
      • Should use delta based on some mix of both parent's error scores
      • re MiniNetManager
      • re RnnSimpleManager
    • Mod app and test code in spec_bench/ai4cr/neural_network/rnn/rnn_simple_manager_spec.cr re training a team of RNN nets on a text file.
      • Mod text util so that when converting float bits to chars, it uses:
        (a) 0.0 when <= 0.0
        (b) 1.0 when > =1.0
        (c) rounds bits [to 0.0 or 1.0]
      • Add #(un)certainty methods for iod data
      • Add code/tests/benches for utf text files (for RNN usage)
      • test: errors should decrease (kinda depends on net and training data structure/size)
      • update this app and shards to be Crystal v1.0 compatible
  • Implement Bi-directional RNN (i.e.: RnnSimple pulls from inputs and previous time column.)
  • Switch to (or add associated classes to use) BigFloat instead of Float64
    • Reasons:
      (a) Outputs are getting too big when using RELU for larger networks.
      (b) I tried handling nan and infinity values via forcing to 0,1,etc or by breeding and purging, but that starts to fail for larger networks.
      (c) I tried auto-scaling down the initial weights (based on network params) to avoid (a), but when scaled down too much (i.e.: larger networks), I get arithmetic errors.
      (d) The things I have tried (or upgrade to Crystal 1.0) seems to now tend to lead to more memory usage (sometimes maxing out my memory).
  • Refactor Chain?); e.g.:
  • Fix spec_bench (re Chain?); e.g.:
    --------
      plot: ''
      error_stats.history: '[]'
    
  • CI should test builds for:
    • Linux
      • Ubuntu (x86) (from the beginning!)
      • Alpine (aarch64)
    • Mac
    • Windows
  • Update https://drhuffman12.github.io/ai4cr/
  • (?) Phase 5: Add more examples and benchmarking (e.g.: simple wave form learning)
  • WHY is the last ti's value in outputs_guessed reset to '0.0' after training (but NOT after eval'ing)??? (and NOT reset to '0.0' after next round of training???)
  • (?) Convert RnnSimple's validation @errors to Hash(<Enum>, String). (I switched it from Hash(Symbol, String) to Hash(String, String), due to issues w/ #from_json. An Enum would probably be more efficient than a String.)
  • Convert all specs to Spectator format.
  • Likewise, split up Chain into chain_concerns and fix where applicable.
    • Check for cleanup in app and tests
  • (re)Update docs (written to https://drhuffman12.github.io/ai4cr/)
  • Code cleanup!
  • Update version!!!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
RNN initially simple RNN
Projects
None yet
Development

No branches or pull requests

1 participant