Hue zigbee manuSpecificPhilips2.state now fully reversed. Several errors and inaccuracies in src/lib/philips.ts found

[chrivers]

Hello Koenkk, et. al. :)

It's been difficult to figure out the best way to report this. I'm not sure if you'd prefer many small issues, a discussion, or one big detailed issue. So here we're trying the latter option. Let me know if you'd like some other way of attacking this thing 👍

It's @chrivers here from the Bifrost project.

Bifrost is a Hue Bridge emulator, which uses one or more Zigbee2Mqtt servers as the backend for zigbee communication.

I've recently been working to add support for more advanced features of Hue lights, including:

• Gradient light support
• Effects ("Candle", "Fireplace", etc)
• Colorered effects
• Animation speed for effects
• etc

Unfortunately, only a small subset of these features are currently exposed through zigbee2mqtt.

Looking at ZHC (src/lib/philips.ts), I was able to get a head start with the current understanding of the zigbee frame format.

After following your excellent guide to sniffing zigbee traffic, I was able to capture a large variety of sample messages, sent between a Philips Hue Bridge, and my LCX005 ("Hue Gradient strip for PC").

At that point, all I needed to do, was to spend a really intense amount of time staring at numbers... 😅

But, good news!

I have managed to completely reverse engineer this format. I'm fairly confident that this is the most comprehensive, correct, and detailed open-source specification of the format available anywhere:

https://github.com/chrivers/bifrost/blob/master/doc/hue-zigbee-format.md

Given the awesome flexibility of this manufacturer-specific frame format, I can see a number of major and minor problems and inaccuracies in the current src/lib/philips.ts converter:

1. Gradient colors are handled as XY-coordinates (a device-independent color space) by the device, but presented as RGB hex values in z2m (a device-dependent color space, with undefined gamut)

2. The (forced) color conversion to RGB is implemented incorrectly.

For example:
"Red" as "Color (X/Y)" Produces a nicely saturated red color
"Red" as "Color (H/S)" Produces the same result
"Red" as a gradient color gives a washed out pink

This is not because the gradient strip cannot show red in gradient mode. For example, sending the command 430101200a200000000e7f5f0e7f5f3800 will produce a deep, saturated red color in gradient mode.

3. The transition time cannot be specified

Sure, the generic "transition time" (right name?) can be specified, but the hue format allows transitioning to a new set of parameters in a unified way. For instance, selecting a "purple, 80% bright, slowly animated fireplace" could be transitioned into with any speed.

4. Gradient scale and offset cannot be specified

These parameters are not available (see specification)

5. Effects speed cannot be specified

This parameter is not available (see specification)

6. Several effects are missing

Here is the complete list of available effects:

Name	Value
NoEffect	0x00
Candle	0x01
Fireplace	0x02
Prism	0x03
Sunrise	0x09
Sparkle	0x0a
Opal	0x0b
Glisten	0x0c
Underwater	0x0e
Cosmos	0x0f
Sunbeam	0x10
Enchant	0x11

Of course, support may vary, but several of these are not supported on any hue device, as far as I can tell.

7. The "gradient style" field is not supported.

The hue gradient strips offers three gradient styles:

pub enum GradientStyle {
    Linear = 0x00,
    Scattered = 0x02,
    Mirrored = 0x04,
}

These produce markedly different results.

8. Updates to EP 11 -> manuSpecificPhilips2 -> Attr state are consumed by the converter

Okay, so this one I'm not 100% sure about. I think that state updates are consumed by this code block:

/* src/lib/philips.ts:434-445 */
     gradient: {
         cluster: 'manuSpecificPhilips2',
         type: ['attributeReport', 'readResponse'],
         convert: (model, msg, publish, options, meta) => {
             if (msg.data && msg.data.state !== undefined) {
                 const input = msg.data['state'].toString('hex');
                 const decoded = decodeGradientColors(input, {reverse: true});
                 if (decoded.color_mode === 'gradient') {
                     return {gradient: decoded.colors};
                 }
             }
             return {};

If my understanding is correct, state updates coming back from the device are consumed (incorrectly), and a lossy conversion from XY to RGB colors for the gradient colors happens, leaving only a fraction of the original, useful data available for user programs.

I would be very happy if the entire (unaltered) reported state attribute would be made accessible in the device state for any device with a manuSpecificPhilips2 cluster.

This would allow z2m clients such as Bifrost to quickly, precise and completely decode the entire device state in a single message.

Well, this turned out to be a long report. I hope you find it useful and reasonably easy to follow. I'm more than happy to answer any questions you might have, and I'm very keen to discuss the possible ways to fix/improve the situation.

You can catch me on discord here: https://discord.gg/YvBKjHBJpA

[danieledwardgeorgehitchcock]

Happy to help out where I can here... I have multiple bulb types to try (no gradients) and some experience of writing complex converters...

[chrivers]

@danieledwardgeorgehitchcock you seem like the perfect candidate for the job 😆

[Mar1usW3]

Nice work. Could also help with a large Festavia which supports all the effects/gardients etc.

[chrivers]

For anyone interested: I've talked to @danieledwardgeorgehitchcock, and he's working on some good improvements to z2m based on my reversing, but we're taking it piece by piece.