@Emil So I wired up a TAPO (TP Link) switch to see if it is advertising _matter. After connecting to WiFi (and before connecting to a Matter hub) I do see _matter._tcp and _matterc._udp. Now let me add it to Flic and see if keeps advertising...
So I added it to my Flic hub, and now it is NOT advertising either _matter protocol. Looks like the proper operation (based on devices from two different manufacturers) is to only advertise when you are not paired with a hub (or maybe in setup mode?). So my speculation here is that it is something with the matter hub- the hub thinks the device is still configured (but shows offline), and the device isn't advertising _matter because it is already owned by a hub...
Let me know if there's anything I can do to continue troubleshooting. I've kept the one device as showing "offline" for the time being...