When you're in the Task, click the little cog wheel on the top left. It will bring up My Task Settings and you can select remove at the bottom of that window that pops up. It will make sure you want to remove it and give you a warning about the task being lost.