Theoretical Question: Control a Zigbee device from outside of network

[washcroft]

Theoretical question...

Armed with an existing Zigbee 3.0 PAN/EPAN, Network Key and Device Address - is there anything stopping sending network packets directly to that device and the device understand/process the commands?

The scenario is that I have an existing Hue Bridge and Hue Bulbs, I want to keep the Hue Bridge as the coordinator of the network, but I also want the ability to control the bulbs directly from an external system (I cannot use the Hue Bridge API for this).