Have lights respond to music

[dan47bennett]

This is a long term goal for the HO, creating functionality to have lights respond to the beat of music. There are several issues to overcome before the final production:

• Access the audio being produced by the PC.
• Extract the frequency content from the audio.
• Work on changing the state of the lights as quickly as possibly.
• Choose how to have the lights respond e.g. brightness, on/off, colour changes etc.
• Reduce latency as much as possible to give instantaneous response.

Latency is obviously hugely important for real-time response from the lights. There is a possibility to produce the audio and introduce a small delay in playback to give synchronous response from the lights, however this is not ideal. The Home API should allow us to avoid this by hosting the network locally.

There are several libraries that can be used to extract the frequency spectrum of audio such as p5.js https://p5js.org/examples/sound-frequency-spectrum.html.

My initial thought is that taking the average amplitude of frequencies below around 100-150Hz should give the beat of the song. Electronically produced music tends to have fairly sparse sounds at this level, usually just a kick which keeps the tempo. Experimentation with what parts of the spectrum to use and how to use multiple lights will be interesting and allow for a lot of customisation.

[fitzhavey]

This is awesome, as we expand into new areas of research - perhaps we are interested in using the wiki section to create a sharedm re-usable knowledgehub

[dan47bennett]

Currently an issue with the home api:
Sending the following request to the localhost server allows the user to change the on and brightness variables, however changing the colour has no effect.

{
    "light": {
        "meta": {
            "hueId": "1"
        },
        "state": {
            "on": true,
            "colour": "rgb(0,220,0)",
            "brightness": "100"
        },
        "name": "danlamp",
        "type": "hue",
        "_id": "ehlg6A207F1uuRhECtlw"
    },
    "network": {
        "ownerId": "zf8aPHxxyffb6v1Ma1OGDynXNhZ2",
        "settings": {
            "lights": {
                "hue": {
                    "username": "pZZSkYfviBrTH4qxNq36Vsur5lYiABEFCsrBLjjs",
                    "ipAddress": "192.168.0.156",
                    "usingLocalAPI": false,
                    "useIFTTT": false
                }
            },
            "IFTTT": {
                "key": "i52Xme7UTQLy7liEykqeF5bwHWcf7Ji3jJKVC7nSzhY"
            },
            "localUrl": "http://localhost:4000",
            "usingLocalAPI": true
        },
        "name": "Dan's Room",
        "_id": "18InBMu296ZOs819j5mu"
    }
}

Clearly the request is being passed to the lights but it is not accepting the rgb values.

[dan47bennett]

After some difficulties it seems that any node packages we attempt to use for this cannot be used within the home API as they are designed for use within a browser. If we used a web browser for sound we would lose the super fast connection the API gives us. Some decisions therefore must be made regarding whether we develop our own way of processing sound or if we decide to integrate the sound processing with the orchestrator site.