Starting with Mutt 1.5.7, you can have Mutt cache IMAP headers to improve IMAP performance considerably. Set the header_cache option in your ~/.muttrc:

set header_cache=/home/bob/.cache_dir