If you click on the video in the "watch later" list, then click on the "Saved" button in the lower left corner of the video, it will change to "Later". That will remove it from your "Watch Later" list.