Fix use-after-free after removing CAs
    
    When removing a CA dynamically, shut its in-progress actions down
    properly, and clean up the state for its actions.
    
    Reproducer provided by Keenan Brock.