They came right off. Here is a pic of two of them.
Okay, so they are branching, squishy, and green. This doesn't match any tube-dwelling worms I'm aware of.
There are no visible polyps indicating hydrozoa either.
Sponges that grow in branches usually have visible holes throughout the structure and are hollow. Example:
https://reefbuilders.com/wp-content/blogs.dir/1/files/2019/10/sponge-coral-reef-3.jpg
If those filaments aren't hollow it just looks like algae

weird.
I'm struggling to identify what kind of algae though. The filaments are really thick and I can't make out what these things are: