Resolves #118: text to speech #119

Baret · 2020-10-20T19:13:42Z

I stumbled upon a text-to-speech library and thought about letting it read out the radio calls. And well, a little tinkering and... here we are :)

branch for #118

...just for fun :P

Baret · 2020-10-26T16:19:55Z

Please have a review (before I can't stop myself and put even more in this "just trying smth real quick"-issue :D).
The different mary versions are required because in 5.2 the rate effect does not accelerate the voice output (aka the effect does not work).

Unfortunately currently the game is only startable from the IDE. When building with maven and starting the application-0.2.0-SNAPSHOT-jar-with-dependencies.jar you get a stacktrace:

Exception in thread "main" java.lang.ExceptionInInitializerError
        at de.gleex.pltcmd.game.application.Main.run(main.kt:58)
        at de.gleex.pltcmd.game.application.MainKt.main(main.kt:43)
        at de.gleex.pltcmd.game.application.MainKt.main(main.kt)
Caused by: marytts.exceptions.MaryConfigurationException: Cannot start MARY server
        at marytts.LocalMaryInterface.<init>(LocalMaryInterface.java:66)
        at de.gleex.pltcmd.game.ui.sound.speech.Speaker.<clinit>(Speaker.kt:79)
        ... 3 more
Caused by: java.lang.NullPointerException
        at marytts.util.MaryRuntimeUtils.ensureMaryStarted(MaryRuntimeUtils.java:70)
        at marytts.LocalMaryInterface.<init>(LocalMaryInterface.java:64)
        ... 4 more

:(

Hint: for proper testing/debugging you should adjust the logback.xml:

    <logger name="de.gleex.pltcmd.game.ui" level="INFO"/>
    <logger name="de.gleex.pltcmd.game.sound" level="DEBUG"/>

game/options/src/main/kotlin/de/gleex/pltcmd/game/options/GameOptions.kt

Baret · 2020-10-28T09:37:34Z

Done.

BTW as a future improvement I thought about "randomized effects". To not make it too monotonous have slight variations in the used effects for every voice line, for example adjust the rate to have slightly faster or slower replay. Or even better: Derive some pitching or equal effect from the callsign so that every element has "its own unique voice". But that is something we can add later.

That is the reason why the filename consists of the text and the effect hash. Actually the speech example first had the option to set different effects to directly compare how they sound on the same text. And it always generated one wav file for a voice line and changing the effect did nothing :D

Loomie · 2020-10-31T08:25:20Z

game/sound/src/main/kotlin/de/gleex/pltcmd/game/sound/speech/Speaker.kt

+import javax.sound.sampled.SourceDataLine
+
+/**
+ * Use this object for text-to-speech. All strings given to [say] wll be read aloud one after another. This means


Suggested change

* Use this object for text-to-speech. All strings given to [say] wll be read aloud one after another. This means

* Use this object for text-to-speech. All strings given to [say] will be read aloud one after another. This means

Loomie · 2020-10-31T08:35:47Z

game/sound/src/main/kotlin/de/gleex/pltcmd/game/sound/speech/Speaker.kt

+
+            playLoop = GlobalScope.launch {
+                log.debug("Launching play loop")
+                while (isActive) {


This seems to be always true as "Stopped play loop" is never logged! I think an own scope is needed that can be cancelled.

Loomie · 2020-10-31T08:54:27Z

game/sound/src/main/kotlin/de/gleex/pltcmd/game/sound/speech/Speaker.kt

+            log.debug("Ignored phrase detected, not saying '$text'")
+            return
+        }
+        val speechDirectory = File(FOLDER_SPEECH_FILES)


File is soo before Java 7. Path and Files is the new I/O 2.

But kotlin.io doesn't seem to know that!?

Loomie · 2020-10-31T08:57:05Z

game/sound/src/main/kotlin/de/gleex/pltcmd/game/sound/speech/Speaker.kt

+
+        // reuse existing texts if they had the same effects
+        mary.audioEffects = effects.toString()
+        val path = "${speechDirectory.absolutePath}/${text.hashCode()}_${mary.audioEffects.hashCode()}.wav"


Please, use a model class (File/Path) instead of String! So you don't have to use the os specific path separator '/ '.

Loomie · 2020-10-31T08:57:56Z

game/sound/src/main/kotlin/de/gleex/pltcmd/game/sound/speech/Speaker.kt

+
+        // reuse existing texts if they had the same effects
+        mary.audioEffects = effects.toString()
+        val path = "${speechDirectory.absolutePath}/${text.hashCode()}_${mary.audioEffects.hashCode()}.wav"


The hashCode() method returns the same hash only in the same JRE runtime instance! It is not fix for multiple application startups!

They are deleted at (normal) JVM-termination anyways ;)

Loomie · 2020-10-31T09:04:11Z

game/sound/src/main/kotlin/de/gleex/pltcmd/game/sound/speech/Speaker.kt

+        }
+        soundFile.deleteOnExit()
+
+        filenames.send(path)


Use soundFile instead of path as String is not very useful.

Loomie · 2020-10-31T09:08:02Z

game/sound/src/main/kotlin/de/gleex/pltcmd/game/sound/speech/Speaker.kt

+            log.debug("Starting MaryTTS engine...")
+            mary = LocalMaryInterface()
+            if(Mary.currentState() == Mary.STATE_RUNNING) {
+                log.debug("MaryTTS started.")


As you mention in the KDoc that the startup may take some time, it would be nice to log the time it took. For me currently the difference between the two log messages is about 1.500 ms.

Loomie · 2020-10-31T09:08:46Z

game/application/src/main/kotlin/de/gleex/pltcmd/game/application/main.kt

@@ -50,6 +55,8 @@ open class Main {
    open fun run() {
        val application = SwingApplications.startApplication(UiOptions.buildAppConfig())

+        Speaker.startup()


As this takes some time consider calling this asynchronous!

Loomie · 2020-10-31T09:10:42Z

game/sound/src/main/kotlin/de/gleex/pltcmd/game/sound/speech/Speaker.kt

+    private fun play(filename: String) {
+        val soundFile = File(filename)
+        if (soundFile.exists()) {
+            val audioStream = AudioSystem.getAudioInputStream(soundFile)


We should check if reading files is faster than calculating the text-to-speech. Because disk i/o is relatively slow. But we could also cache the bytes in memory (for example with a fixed 10 MiB pool). But this are performance considerations.

Loomie · 2020-10-31T09:25:01Z

game/sound/src/main/kotlin/de/gleex/pltcmd/game/sound/speech/Speaker.kt

+
+                    it.drain()
+                }
+            } finally {


Looks like you want to lock the audio when in use. So instead of an AtomicBoolean I would suggest a corresponding mutual exclusion functionality from java.util.concurrent. Maybe a Semaphore is more approtiate as a Lock. The benefit is that you can just wait for it to be freed instead of the while-delay in waitForQueueToEmpty().

Edit: This looks like a Guarded Block.

PS: maybe we can just use Jobs? They can be waited for (joined).

game/sound/src/main/kotlin/de/gleex/pltcmd/game/sound/speech/Speaker.kt

Loomie · 2020-10-31T09:34:47Z

game/sound/src/main/kotlin/de/gleex/pltcmd/game/sound/speech/Speaker.kt

+                    it.start()
+
+                    var count = 0
+                    val buffer = ByteArray(4096)


Why 4096 instead of sourceDataLine.getBufferSize()?

Loomie · 2020-10-31T10:07:14Z

game/sound/src/main/kotlin/de/gleex/pltcmd/game/sound/speech/Speaker.kt

+@ExperimentalCoroutinesApi
+object Speaker {
+
+    private const val FOLDER_SPEECH_FILES = "./speech"


remove the os specific "./" and just use the name. Or provide a path object (see below)

Loomie

Looks and works good! For some implementation detail improvements see the comments.

…peech # Conflicts: # game/application/src/main/resources/logback.xml # game/ui/src/main/kotlin/de/gleex/pltcmd/game/ui/views/GameView.kt

# Conflicts: # game/application/src/main/resources/logback.xml

Baret added 12 commits October 19, 2020 22:35

playing around with maryTTS

0a75e09

branch for #118

use male voice

d757695

Use faster voice for speaker, created example main

e85ad49

Added ignored phrases, initializing effects correctly

3903c57

Made sound optional

bf9a086

Added exclusions to continue using voice 5.2

0d52448

Ignore log and speech folders

1dc3d5d

fix for enforcer rules regarding commons-io

651d0c7

Moved Speaker to new module sound

d0fb397

Added proper startup and shutdown for Speaker

0724d0a

Built model classes for effects

1fce8cc

...just for fun :P

Added unit test and documentation

0e28db7

Fixed package name

ba94754

Baret requested a review from Loomie October 26, 2020 17:06

Loomie reviewed Oct 27, 2020

View reviewed changes

game/options/src/main/kotlin/de/gleex/pltcmd/game/options/GameOptions.kt Outdated Show resolved Hide resolved

Use upper case for options constant

e41a3ac

Loomie reviewed Oct 31, 2020

View reviewed changes

game/sound/src/main/kotlin/de/gleex/pltcmd/game/sound/speech/Speaker.kt Show resolved Hide resolved

Loomie reviewed Oct 31, 2020

View reviewed changes

Loomie assigned Baret Oct 31, 2020

extracted function

5cdd54f

Baret added the feature A completely new feature label Nov 26, 2020

Baret linked an issue Nov 26, 2020 that may be closed by this pull request

Text to Speech #118

Open

Merge remote-tracking branch 'origin/master' into issue-118-text-to-s…

05ebea4

…peech # Conflicts: # game/application/src/main/resources/logback.xml # game/ui/src/main/kotlin/de/gleex/pltcmd/game/ui/views/GameView.kt

Baret added 2 commits January 5, 2021 11:45

Merge branch 'master' into issue-118-text-to-speech

684168c

# Conflicts: # game/application/src/main/resources/logback.xml

re-added jcenter repo

5275638

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resolves #118: text to speech #119

Resolves #118: text to speech #119

Baret commented Oct 20, 2020

Baret commented Oct 26, 2020 •

edited

Loading

Baret commented Oct 28, 2020

Loomie Oct 31, 2020

Loomie Oct 31, 2020

Loomie Oct 31, 2020

Loomie Oct 31, 2020

Loomie Oct 31, 2020

Baret Oct 31, 2020

Loomie Oct 31, 2020

Loomie Oct 31, 2020

Loomie Oct 31, 2020

Loomie Oct 31, 2020

Loomie Oct 31, 2020 •

edited

Loading

Loomie Oct 31, 2020

Loomie Oct 31, 2020

Loomie Oct 31, 2020 •

edited

Loading

Loomie left a comment

	* Use this object for text-to-speech. All strings given to [say] wll be read aloud one after another. This means
	* Use this object for text-to-speech. All strings given to [say] will be read aloud one after another. This means

Resolves #118: text to speech #119

Are you sure you want to change the base?

Resolves #118: text to speech #119

Conversation

Baret commented Oct 20, 2020

Baret commented Oct 26, 2020 • edited Loading

Baret commented Oct 28, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Loomie Oct 31, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Loomie Oct 31, 2020 • edited Loading

Choose a reason for hiding this comment

Loomie left a comment

Choose a reason for hiding this comment

Baret commented Oct 26, 2020 •

edited

Loading

Loomie Oct 31, 2020 •

edited

Loading

Loomie Oct 31, 2020 •

edited

Loading