This is a funny one that seems to trick a lot of people… After clicking delete, you then get an AJAX message at the top of the screen asking you to confirm deletion. You have to click yes at the top of the screen after deleting the AD Access group at the bottom of the screen. Bad UI design 😉
If you have an existing message from a previous event (such as a delete), you will have to close the message at the top of the screen (cross in right hand corner) before you can see the delete confirmation of the next message.
Once you accept these, the AD Access group will then be deleted.