Google Chrome 58 making NET::ERR_CERT_COMMON_NAME_INVALID all over the place

We use self signed certificates at LeadSift for our dev and qa servers behind the firewall and have our team just import the certificate into their computers. Recently after months of it working just fine, I got an error on Google Chrome which looked like this.

Attackers might be trying to steal your information from site.dev.leadsift.com (for example, passwords, messages, or credit cards). NET::ERR_CERT_COMMON_NAME_INVALID

This server could not prove that it is site.dev.leadsift.com; its security certificate is from [missing_subjectAltName]. This may be caused by a misconfiguration or an attacker intercepting your connection. Learn more.