Jarbas is a fork of mycroft-core
everything can break anytime! things designed for mycroft may not work with jarbas! some things designed for jarbas wont work in mycroft! support for this branch is limited to me alone, folks from mycroft community may not be able to help
that said, everything that works in mycroft i try to make work with jarbas, but i give absolutely no warranty of this!
- bus address is different, but it is supposed to be configured in config file, that means the 3rd party should support it and not hard code it
- msm temporarly removed, skill install is old-school manual install
- no use of mycroft.ai, this means no web ui (yet) and harder setup
- no online config, must edit files
- no mycroft stt, must configure Kaldi or other online service
- no shared api keys (weather/wolfram...)
- some skills not fully functional / added yet / configured for server / undocumented setup
Still interested?
This branch (patch-15) is a work in progress, forever in development, the other branches should all be considered broken / abandoned / temporary
Check the jarbas_docs folder for more info on each change, expect this to be outdated
This should install in a virtual env just like mycroft, only debian install scripts were updated
virtual env is named jarbas , it is possible to have both mycroft and jarbas at same time, each one running in a different virtual env and using different bus adress
whenever official mycroft documentation mentions "workon mycroft" use "workon jarbas"
not everything will work out of the box, expect bumps
Personal assistants today are glorified voice activated script runners, with the bonus of mining your data for some company
Stick some hardware and they are glorified alarm clocks than run scripts instead
Jarbas fork intends to show that mycroft is much more than this by harnessing the power of open source, an AI assistant should do "AI stuff" on demand, where is the AI in mycroft/jarbas?
#### TODO links in each one ####
The essential parts that make it work, many options available for each
- speech to text
- text to speech
- intent from text (text to action)
The "AI stuff" it can already do for you
- Image classification
- Object recognition
- Face recognition
- Voice recognition (wip)
- Deep Dreaming
- Generate names with Restricted Boltzman Machines
- Generate Poetry with MarkovChains
- Copy the style of any artist using Style Transfer
- Colorize Black and White photos
- Chatbot using AIML
- Auto complete your stories with a Char-RNN (training...)
- Learn and build a database of what it learns with LILACS
And dont forget the "not really AI" stuff
- post to facebook / twitter / instagram
- control a browser like a real human and do any kind of stuff with it
- generate psychedelic images using mathematics
- search the internet for answers
- Search and play music from youtube (or spotify or more places!)
- Stuff not worth mentioning like parrot back, dictation, tell jokes, hows
the weather, which asteroids are near earth, how many sunspots currently
visible, show you the latest image of earth from orbit..
In practice it can act as a vocal search engine, a helper for disabled people, a desktop automation/OS framework, an artificial pet, a Art assistant, a Writers Assistant, an educational toy for children, A work of art by its own merit...
- no pairing
- no config update from mycroft.ai
- added "intent.start", "intent.end", "intent.failed" messages
- skills/core.py SKILLS_DIR is now jarbas_skills folder instead of opt but configurable
- do not remove articles in normalize (too much normalization IMO)
- allow for multiple hot words of various kinds
- offline stt (pocketpshinx) <- WIP
- wakeword/hotword/standupword snowboy support
- fallback skill order override in config file
- apis centralized in self.APIS for all skills
- message_context update methods added to MycroftSkill
- update_config method added to MycroftSkill
- screen display service
- speech messages transmit metadata together with speech
- speech messages set adapt context for LastSpeech and every metadata field
- recognizer_loop:speak / hotword / external stt message types
- save utterances if desired (configurable, disabled by default)
- utterance and hotword save paths configurable
- external STT requests (transcribe audio from a skill)
- runtime shutdown/reload of skills
- runtime changes of configuration (tts/stt/hotwords)
- centralized API section in config
- ConfigurationManager.save checks for config existence before loading it
- webchat client
- encrypted multi instance networking (server/client)
- multiple connected clients support, authorization per client per action
- many helper classes
- many skills bundled by default
# temporary changes
- no msm
Display Service Demo
Hotwords/Snowboy Demo
TTS runtime change Demo
Facebook chat client Demo
LILACS learning demo
Morse Code TTS engine joke
AIML chatbot demo
only debian officially supported, ubuntu complains because firefox package name is different
- build_host_setup_debian.sh
- build_dev_setup.sh
if you don't care about pre-packaged skills / using a pi
- build_host_setup_debian_light.sh
- build_dev_setup_light.sh
this won't install most skill requirements, only core and client/server requirements
in the future will install pymimic instead of mimic
to update packages:
- git pull -> for merging this fork changes
- re-run host setup script -> for updating system packages
- run dev-update.sh or dev-update-light.sh -> update python packages in virtual env
using full update scripts:
this means overwrite all changes and revert to current github state, update host packages, and update dev packages
- full_update_debian.sh or full_update_debian_light.sh
to start it is the same as mycroft-core, jarbas.sh and mycroft.sh are duplicates
- TODO - all usage flags -c -v -cl -s -w
- ./jarbas.sh start
or with different processes
- ./start.sh service
- ./start.sh skills
- ./start.sh audio
- ./start.sh display
- ./start.sh voice
- ./start.sh cli
- ./start.sh server
- ./start.sh client
- ./start.sh webchat
all others scripts are not meant for use, open them to understand what they do, comments explain most things
of interest:
- pymimic install scripts
- tensorflow model download scripts in jarbas_models/model_folder
- all mimic voices download script
IntentLayers class, allows to activate and deactivate intents grouped in layers
Example usage in KonamiCode Skill
Backend Service, A helper class to emit and wait for bus messages was added, needed in several skills
Used for inter skill communication in image recognition, face recognition, object recognition, deep dream, style transfer, vision, user manager...
Markov Chain helper class, save and load functionality
Example usage in Poetry Skill
Generate random pictures utility based on this blog post
Example usage in Art Skill
IntentParser class
- helper class added to intent_service
- intent from utterance
- skill from intent
- skill from utterance
priority skills list
- priority skills are loaded first
- read from config
needed for intent_layers
- enable / disable intent can be requested in messagebus
control skills externally, allow run-levels, external skill load and reload)
- skills have internal id, internal id used instead of name in intent messages
- skills main now listens for external messages for skill_manifest, skill_shutdown and skill_reload
- uses message context to target messages to source of utterance
- uses message context to mute (not trigger tts)
- uses message context to tell if more speech is expected
- adds "metadata" field to speak method
- clients check if they are the target of utterance before processing it
- adds data to message context:
- target = cli/speech/facebook - default = "all"
- mute = trigger tts or not - default = "false"
- more_speech = more speak messages expected- default = "False"
- destinatary = used to track remote_client to send message to
- source = source of message, default = source skill / source client
- adds disable_speak_flag to speech and cli
- this flag is a global mute switch, if set no tts is ever made
- skill to mute/unmute not yet merged
- client / server (a "firewall" that connects to the mycroft messagebus )
- client connects to server, by secure websockets
- server sends public pgp key
- client sends public pgp encrypted with server pgp
- server sends AES key encrypted with client pgp, client pgp key authorizes client / creates account in server
- all communication after is AES encrypted, if plaintext received client is kicked
- server does all kinds of processing (receiving files, requesting stuff from clients, authorizing action per client)
- on first run a new pgp key is automatically created (client and server)
client server basically connect jabas/mycroft instances, server can work as:
- fallback mechanism
- chat room
- relay
- ...
Lots of skills pre-packaged
TTS Control - change voice at runtime vocally
Control Center - shutdown/restart skills , enforce run levels
Vision Service - process webcam input
RBM Service - Generate new words
Dictation - Write to txt user speech
News - Plays news
Mute - Mute TTS
Again - Repeat last action
LILACS - Search wikipedia, wikidata, dbpedia, conceptnet, wordnik, wolframalpha and learn
TODO finish this
some skills are blacklisted because they are either not common, heavy, or because i run them in server
- browser service
- server fallback
- LILACS_core
- LILACS_storage
- LILACS_teach
- LILACS_users
- LILACS_curiosity
- deep dream
- image recognition
- object recognition
- colorize
- vision
these skills have a flag in their init method to ask server or run locally
consider them broken, get them from server branch if you really want them
- image recognition
- face recognition
- object recognition
- deep dream
- style transfer
- colorize
these skills have a flag to ask client
- vision service # wip
not updated often
- Help Skill
- R2D2 TTS
- instagram skill
- pocket sphinx offline stt
- kaldi install scripts
- class vizualization/deep draw in tensorflow
- models download script (they auto download on first skill run which takes...long)