Annelid fossils include tracks and burrows, jaws, and whole bodies. They were already diverse in the Cambrian fossils of the Burgess Shale. One of them called Canadia is much like a modern polychaete worm with a fairly complex brain. Modern annelids with simple brains evolved from ancestors with more complex brains.
Ancestral annelids had a reasonably complex nervous system that has been reduced numerous times independently during transitions in lifestyle and/or extensive modifications of the annelid body plan.
Annelids were even burrowing in the sediment during the Cambrian.