Okay, so all of the angels vamoosed and left the universe, an action which removed all power from Castiel.
...so then why do two angels still have power?:
* future!Lucifer still has the teleportation/relocation thing working...and the impression I got from future!Dean's words was that Sam became a vessel recently -- which means it happened after the other angels left.
* present!Zacharia was able to appear out of nowhere and relocate present!Dean to the present -- so clearly nothing was sucking out his mojo.