(a)
http://stackoverflow.com/questions/10803671/which-format-to-use-for-the-cn-dns-fields-in-a-ssl-certificate-for-an-idn

(b)
http://tools.ietf.org/html/rfc6125
Representation and Verification of Domain-Based Application Service
Identity within Internet Public Key Infrastructure Using X.509 (PKIX)
Certificates in the Context of Transport Layer Security (TLS)

6.4.2. Checking of Internationalized Domain Names

   If the DNS domain name portion of a reference identifier is an
   internationalized domain name, then an implementation MUST convert
   any U-labels [IDNA-DEFS] in the domain name to A-labels before
   checking the domain name.  In accordance with [IDNA-PROTO], A-labels
   MUST be compared as case-insensitive ASCII.  Each label MUST match in
   order for the domain names to be considered to match, except as
   supplemented by the rule about checking of wildcard labels
   (Section 6.4.3; but see also Section 7.2 regarding wildcards in
   internationalized domain names).

U-label: The Unicode representation of an internationalized domain name,
i.e. how it is shown to the end-user.

A-label: The ASCII-compatible encoded (ACE) representation of an
internationalized domain name, i.e. how it is transmitted internally
within the DNS protocol. A-labels always commence with the prefix
“xn--”.

(c)
https://wiki.mozilla.org/CA:Recommended_Practices#Document_Handling_of_IDNs_in_CP.2FCPS
(search for IDN)

I've checked the Firefox source code, and as far as I can tell (call chains below) the code always ends up calling SSL_SetURL with the ASCII form, and NSS doesn't do any conversion when matching the contents of this field with the certificate.  So this question seems settled, both producers and consumers of the data use the ASCII form.
    Mirek


{nsHttpChannel::BeginConnect(),nsHttpConnectionMgr::ReportFailedToProcess(uri),nsHttpHandler::SpeculativeConnect(aURI, _)}
# All callers above use URI->GetAsciiHost(), i.e. ACE
-> nsHttpConnectionInfo::nsHttpConnectionInfo(host, ...)
   -> nsHttpConnectionInfo::SetOriginServer(host, _)
      => nsHttpConnectionMgr::nsHalfOpenSocket::SetupStreams(...)
     # mEnt->mConnInfo->Host uses types nsConnectionEntry->nsHttpConnectionInfo
     -> nsSocketTransportService::CreateTransport

nsFtpState::EstablishControlConnection
# uses mChannel->URI()->GetAsciiHost(), i.e. ACE
-> nsFtpControlConnection::nsFtpControlConnection(host, _)
   => nsFtpControlConnection::Connect # nsCString mHost
      -> nsSocketTransportService::CreateTransport

nsFtpState::R_pasv()
# uses either NetAddrToString == IP[46] address
# or mChannel->URI()->GetAsciiHost(), i.e. ACE
-> nsSocketTransportService::CreateTransport

# seems unused
Dashboard::RequestConnection(aHost, ...)
-> Dashboard::TestNewConnection(aHost, ...)
   -> nsSocketTransportService::CreateTransport

----

nsSocketTransportService::CreateTransport(_, _, host, ...)
-> nsSocketTransport::init (_, _, host, _, _)
   => nsSocketTransport::BuildSocket # nsCString mHost
      -> *:NewSocket
      -> *:AddToSocket

----

{nsTLSSocketProvider,nsSSLSocketProvider}::NewSocket(_, host, ...)
-> nsSSLIOLayerNewSocket(_, host, ...)
   -> nsSSLIOLayerAddToSocket

{nsTLSSocketProvider,nsSSLSocketProvider}::AddToSocket(_, host, ...)
-> nsSSLIOLayerAddToSocket

----

nsSSLIOLayerAddToSocket(_, host, ...)
-> SSL_SetURL(sock, host)