@Emil thanks for your fast response. I tried it quickly and measured around 50 tries with varying results. First of all I used just a speech output from my Echo as you recommended. The normal response was about 2-4 seconds which is absolutely acceptable.
Then I directly talked to my echo speaker to turn my light on and off, this was also very quick with about 2-4 seconds.
I observed sometimes my flic glowed orange and sometimes it took very long to response with both, the speech and the light.
It could be that all devices add a small delay and in total it feels rather slow.