This happens to me as well. If you look at VA, you'll see that you have it set to not listen, but if you move your mouse over it, the red circle with a slash through it will disappear, revealing that it wasn't really turned off, so you click on it to turn it off, and move your mouse over it again, the red circle disappears again, so it wasn't turned off again. Sometimes you have to click randomly on VA, and then try to turn it off again. Just keeping doing that till when you run your mouse over it, the red circle stays there, then it's truly off.
Now, this is something that doesn't happens all the time, but it happens more times then I like. It's just a nuisance bug that happens from time to time.
Just remember to run the mouse over it, to see if the red circle stays or not.