Chromecast works after setup, but does not work later on

[flic1010]

I have connected a Flic button to my Android phone and configured it to jump to the next track for my Chromecast.
When I set this up, then the button action is working.
However when I come back later on, then the button does not work anymore. Clicking the button is noted from my phone, but it is only giving some negative audio feedback.

"Come back later on" means that my phone had been connected to other networks and the Chromecast had been powered off and on again.

Is there any way to make the Chromecast action work all the time?