Policy Review Board Case Summaries

The Policy Review Board examined the grading policy for detections of multiple Potentially Exploited events in a single day. Bitsight’s Compromised Systems section measures outbound detections related to malware (Botnets, Potentially Exploited), spam or other suspicious activity (Spam Propagation, Unsolicited Communications), and servers that attempt to distribute malware (Malware Servers).

Compromised System events are graded based on their frequency and duration. Single-day events are considered to be remediated if no future detections can be observed, and carry the least impact in comparison to events detected over a 3-day span (or longer), with the impact continuing to scale as long as the infection continues to beacon out. Likewise, multiple infections detected on the same day can also have an increased negative effect on the rating.

The Policy Review Board decided not to make a change in policy at this time. Compromised Systems findings are used to draw statistical inference about the strength of any controls that might mitigate infections; for instance, if an antiviral software is lacking up-to-date definitions it may not recognize a malicious file. Additionally, different infections on the same user device may demonstrate different controls issues depending on the malware family.

A rated company raised an issue with how Bitsight grades multiple certificate findings for the same hosts. Upon reviewing the data, it seems the default certificate, "Kubernetes Ingress Controller Fake Certificate", has a 1-day validity period, and it issues a new self-signed certificate each day. As each rescan will produce a new “Warn” finding, it has the potential for a single host to produce dozens of Warn results within the Event Lifetime period, disproportionately affecting the SSL Certificates grade.

The Policy Review Board has decided to not make a change in grading policy at this time. We believe that the self-signed certificates highlight a real risk: a system that was not intended to be publicly accessible, is in fact accessible from the Internet. The PRB agrees that it would be ideal to be able to further analyze these records in the platform to identify the root cause (exposed kubernetes ingress controller, in this case). Bitsight will investigate whether we can add additional context to the finding details. Unfortunately, it’s not technically feasible for us to deduplicate these findings in the short term: we use the certificate itself as a deduplication key, and we don’t have any way to match up “identical” certificates across different days in a consistent manner.

A rated company challenged Bitsight’s grading policy for Diffie-Hellman keys shorter than 2048 bits, as they believed that frequent key rotation could mitigate the risks of a shorter key size. Diffie-Hellman (DH) key exchange is used in SSL/TLS to establish a shared secret key between a client and server. Secrecy relies on the difficulty of inverting a cryptographic operation (discrete logarithm); in general, the longer the prime number p used in the DH key exchange, the more difficult it is to recover the secret key by brute force. A technique called Logjam (discovered in 2015) reduced the difficulty of this operation. It’s believed that recovering 1024-bit DH keys is currently feasible for a nation-state actor. As a result, NIST has recommended since 2015 that DH keys be at least 2048 bits.

The Policy Review Board decided not to make a change in grading policy at this time. The current grading policy that requires 2048-bit Diffie Hellman primes is well supported by recommendations from standards bodies, including NIST. There is no similar recommendation to support the practice of frequently rotating shorter, weaker primes.

A rated company requested that Bitsight make available a way for their organization to review newly discovered assets--and findings related to those assets--before these items could impact their corporate tree. Today, Bitsight rates all organizations based on any publicly discoverable IPs and domains and their configurations as soon as they are identified, and all of these assets must be visible at the top of the ratings tree (although they may originate from a subsidiary). This structure is designed to provide an unbiased view of a rated company’s organizational risk even if they have collaborated with Bitsight to provide additional self-published structure to their tree, which may otherwise alter how their risk is perceived by an observer.

The Policy Review Board decided not to implement a change in policy at this time. Any implementation of the requested review process would become a de facto change in Bitsight’s existing policy requiring all findings and assets to be visible upon discovery in an organization’s infrastructure.

A rated company challenged Bitsight’s grading policy for SSL certificate subject name mismatches. SSL certificates enable secure communication between a client and a server. Among other things, the certificate establishes the authenticity of the server (to prove to the client that the server is who it claims to be, and is not an attacker). To do this, the server hostname must be listed on the certificate. If it’s not, a certificate subject name mismatch error results when the client attempts to establish an SSL connection. This error can be ignored at the user’s discretion, but this opens the client to the risk of man-in-the-middle attacks.

The Policy Review Board decided to not make a change in grading policy at this time. Our policy is to grade cert name mismatches negatively because the TLS/SSL protocol requires a proof of authenticity to ensure the client is interacting with its intended destination -- a core factor in limiting the ability to conduct Man-in-the-Middle attacks. For example, if a client intends to visit a bank’s website, www.bank.com, but a certificate for www.website.com is returned by the server, it cannot trust whether it’s actually talking to www.bank.com. This is known as a certificate mismatch error and is established to be a TLS/SSL configuration error.

In general, we cannot change the rating impact of findings based on evidence of compensating controls; however, we do allow rated companies to annotate the risk or to exclude the asset in question from the company’s primary rating.

A rated company challenged Bitsight’s grading policy for SMTP ports that support SSLv3. Today, Bitsight negatively grades all services that offer SSLv3, even if a more modern protocol is also offered by the server. SSLv3 is a depreciated protocol that, under certain conditions, can be exploited using an attack called POODLE (Padding Oracle On Downgraded Legacy Encryption) which allows man-in-the-middle attacks to recover transmitted secrets. The appeal was introduced due to the need for SMTP servers to offer backwards compatibility for older clients and also due to claims that POODLE is an HTTPS-specific attack.

The Policy Review Board decided not to make a change in grading policy, either retroactively or in the future. The overwhelming majority of security practitioners and standards bodies consider SSLv3 to be obsolete and insecure. Although we acknowledge that the known SSLv3 vulnerabilities may be more difficult to exploit in practice for SMTP connections, the possibility remains; for example, the POODLE padding oracle exists for any connection, not just HTTPS. To this point, there is sometimes confusion due to the fact that the original paper that introduced POODLE used HTTPS as an example of an exploit mechanism, but POODLE is not restricted to HTTPS.

A rated company raised an issue with how Bitsight grades certain certificates. Established best practices dictate that internet-facing assets that serve leaf certificates must also provide intermediate and root certificates to prove their authenticity to the client. Bitsight performs trust anchor validation and negatively grades certificates that are either self-signed or fail to include the other certificates in their chain of trust, even if a client may be able to complete the chain with their own cached certificates. Some TLS/SSL services, however, require a response with a client certificate issued in advance or the server will terminate the connection with a TLS error and prohibit access to the application layer (e.g., a website); these services may be self-signed or otherwise fail the trust anchor validation process.

The Policy Review Board concluded that Bitsight should change policy when possible to exclude certificate chain validation for systems that require client certificate authentication, as the PRB considers it a reasonable mitigation to the risks posed by having self-administered trust anchors.

A rated company appealed Bitsight’s grading of the X-XSS-Protection header in the Web Application Headers Risk Vector. Currently, Bitsight assesses headers that are minimum expectations, referred to as required headers, and those that may be implemented optionally depending on the configuration of the web page. The X-XSS-Protection header is considered an optional header; these headers carry no weight if not included, and if included they can contribute positively or negatively to the impact of the record depending on its implementation.

After deliberation, the Policy Review Board decided to recommend a change in policy when possible to no longer grade this header. This decision was reached because (1) well-configured Content Security Policy implementations can render this header redundant, (2) OWASP now recommends setting the header to zero to disable the X-XSS-Protection auditor, and (3) almost all major browsers have depreciated the header.

A rated company requested to add a joint-venture subsidiary to their company tree. Today, Bitsight requires that subsidiaries are 100% wholly-owned or more than 50% owned if the company is compliant with the Accepted Accounting Standards.

The Policy Review Board decided to recommend a change in policy to allow joint-venture companies to be included under certain conditions. We will consider a company to be a subsidiary of a parent for rating purposes if 1) the subsidiary is wholly owned by the parent company, OR 2) the parent has above 50% ownership and is IFRS-10 compliant, OR 3) both the parent and the subsidiary agree in writing to the relationship.

A rated company raised an issue with an SSL Configuration that had been fluctuating between “Good” and “Bad” due to their Web Application Firewall (WAF). In some cases, Bitsight scans can be blocked by the firewall and served other content, such as a captcha page. If this is the case, successive scans could report different configurations depending on what content is offered.

Our research team thoroughly reviewed this case and determined that there was a clear way to remediate this issue. After research it turns out that the technology in use by the rated company was seen deployed by other organizations with TLS v1.2 and v1.3 enabled, which are currently considered secure. The company had an older version of the WAF that enabled TLS v1.1 which is vulnerable and no longer recommended.

A rated company requested that Bitsight begin to exclude expired certificates from grading that are hosted on assets that they no longer control, if they have ended a contract with that provider. Services that provide certificates that are expired are graded as “Bad”, and can indicate that the current controls an organization implements to manage these certificates needs to be reviewed. However, some hosting providers do not immediately delete client data upon termination of a contract, leaving certificates to expire with potentially little recourse for the affected organization.

The Policy Review Board concluded that Bitsight should implement a policy to remove these expired certificates from reports if it can be confirmed that there is no longer any active association between the provider and the originating company. This change in policy would only affect a rated company if there is no relevant WHOIS or DNS record that would indicate that the certificate is still in circulation.

A rated company requested that Bitsight create an exception for SSL Certificates signed by an internal Certificate Authority (CA). Best practice dictates that certificates that are publicly available must be signed by a public CA, allowing clients to verify that the site is authentic and in good standing. The rated company appealed on the grounds that some VPN gateway services have been observed with internal CA signed certificates and that these services are only intended to be used by a small number of clients.

The Policy Review Board decided not to implement a change in grading policy at this time. Many VPN services support the use of public CA certificates and recommend their use. A primary argument downplaying the risk of a subset of internal CA certificates is that they are only designed to communicate with a specific set of clients; in such circumstances, it is a best practice to prevent the exposure of such interfaces to the general Internet. Self-signed and internally issued certificates are additionally prone to broad categories of weak implementations that are either less likely to occur or outright prevented by public CA signing policies. A key risk uncovered by the SSL configuration risk vector are devices that are initialized and connected to the Internet, yet are never completely set up or are hardened properly (e.g. may have default credentials). Many such instances manifest as network appliances configured with self-signed or default certificates that are not rooted to a valid trust anchor and not designed to be used in production systems.

A rated company appealed Bitsight’s grading of the rdate protocol. Today, rdate is graded as a “Warn” issue and is a protocol designed to respond to requests for the current time. The rated company appealed on the grounds that (1) it felt that the time protocol posed no risk and (2) it has fewer known vulnerabilities compared to other services that are graded NEUTRAL, such as DNS or the alternative Network Time Protocol (NTP).

The Policy Review Board decided not to implement a change in grading policy at this time. This protocol has known security vulnerabilities; If the time is incorrect, it can be exploited by attackers to break secure connections and encryption certificates. This is in line with NIST recommendations, and NIST additionally points out that rdate “has poor error-handling capabilities in general, and many of the client programs that use this format are poorly written and may not handle network errors properly”. Implementations of obsolete protocols can also be a sign of poor overall configuration management.

A government agency requested that Bitsight prevent third parties from subscribing to ratings within its rating tree. At the time this was raised, Bitsight made available to customers any rating that it could generate for an organization. This agency raised concerns that data made available through Bitsight could pose a security risk if sensitive information about configurations were revealed through scans.

The Policy Review Board decided not to establish a policy of restricting subscriptions to ratings of any particular classes of entities, including national infrastructure. However, Bitsight does have a feature, Manage IP Visibility, which, when enabled, will prevent any subscribers from being able to view the infrastructure or IP addresses of findings for that entity. We strictly adhere to a responsible disclosure policy.

A rated company challenged Bitsight’s policy of including all events within a company’s infrastructure in the top-level rating of the company’s corporate tree (The Bitsight ratings platform allows an organization to be divided hierarchically into subsidiaries, but the root of the tree always represents the entire organization.) In this case, the company used a portion of its infrastructure for deliberately running malware for research purposes; the company wished this infrastructure to be excluded. Bitsight does allow each organization to designate a subset of its infrastructure to be included in its primary rating. Customers can choose to subscribe to either the top-level rating, or the primary rating, or both, but are encouraged to monitor the primary rating, as it often better contextualizes the security performance of the organization. However, the appealing company in this case felt that relatively few of its customers were monitoring the primary rating vs. the overall rating, and so wished for the infrastructure in question to be excluded from all parts of its ratings tree.

The Policy Review Board decided not to make a change in policy at this time, in part because of the importance of applying a consistent and objective methodology across the entire inventory of rated organizations, and also the importance of providing a comprehensive view to clients of the rated organization. However, the PRB directed Bitsight’s research teams to investigate ways in which this might be done in the future. In the short term, the PRB recommended additional steps to encourage customers of the appealing organization to monitor the primary rating.

A rated company requested that the ratings impact of a compromised systems event be reduced, on the following grounds: the company felt that there was insufficient forensics information available in the Bitsight portal for the company to locate and remediate the event; therefore, the incident continued for several days and resulted in further ratings impact. The PRB decided not to change the ratings impact of the event; as in similar cases in the past, the PRB reiterated that the responsibility for detecting and remediating security issues must ultimately lie with the company itself. However, the PRB did recommend improving the level of forensics detail available for such events. Additionally, the impact of events spanning multiple days is being revised as part of the ratings algorithm update later this year.

Two rated companies requested a change in policy for evaluating certificates used for virtual private network (VPN) servers. Under current policy, certificates that are not rooted to a well-known public certificate authority are graded negatively. Certain VPN solutions are configured by default to use self-signed certificates; others use internal (non-public) certificate authorities, which prompted the request for the policy change. The Policy Review Board carefully considered this request. As part of the decision process, the PRB and Security Research team investigated several solutions from VPN vendors, including vendor support for certificates rooted to a public certificate authority.

Based on this investigation and the recommendations of the Security Research team, the PRB elected not to make a change in grading policy at this time. Using public certificate authorities is still considered best practice (and is supported by all major VPN vendors); internally rooted certificates pose additional risks. The PRB did recommend updates to the knowledgebase articles on the SSL Configurations risk vector to more clearly explain the rationale for this grading policy.

A rated company made an appeal concerning an update in impact to a breach it had experienced. The ratings impact of a breach depends on the severity of the incident, and in particular the count of records exposed. In this case, new information about the breach emerged approximately six months after the incident was first recorded. Based on these new revelations, Bitsight retroactively increased the ratings impact of the incident. This is in keeping with current policy, although the length of the delay was unusual. The rated organization requested a change in policy to limit the window of time in which security incident impacts can be changed. It also requested that it receive alerts for any changes to past incidents.

After extensive discussion, the Policy Review Board decided not to implement a cutoff on when new information might alter the impact of a security incident that has already been recorded. The PRB weighed the value of such additional information against the effect of these retroactive updates on the customer experience. Adjustments as delayed as this particular one (six months) are quite rare, and even then the change in impact is typically small (eight points in this case). On the other hand, Bitsight has substantial analysis that indicates that one incident is correlated with increased risk of another in the future, and that this correlation holds for multiple years. Taking all of this into account, the PRB decided that it would be difficult to implement a policy that would preserve the accuracy of the rating in these cases.

Regarding the lack of alerts when the impact of an incident is retroactively updated, the PRB agrees that this should be addressed. This enhancement has been added to the official product roadmap.

Several rated companies escalated incidents involving spam propagation findings. The activity in question was detected via a network of sensors deployed across thousands of mail transfer agents (MTAs). The findings were flagged as suspicious because their HELO string did not match the sender’s IP address or domain. This is often indicative of malware used in spam campaigns. However, several rated companies had difficulty in locating the source of these events, in part because there is often a large volume of legitimate email traffic to search through to find isolated anomalies. The companies appealed on two grounds: (1) that their ratings should not be reduced for events which they could not locate, and (2) that the ratings impact was disproportionately large, especially since many events spanned multiple days, and therefore resulted in multiple rating drops. On point (1), the Policy Review Board reiterated that while Bitsight will make every effort to provide information that can help organizations understand and remediate issues, the remediation responsibility ultimately lies with the organization itself, and failure to locate compromised machines or other security issues cannot be considered for ratings purposes. However, the PRB did recommend improvements to the forensics details provided for spam propagation events, and also improvements to the knowledgebase articles on the subject. On point (2) the impact of events spanning multiple days is being revised as part of the upcoming ratings algorithm update; details will be announced later this year.

A rated company requested that Bitsight grant a preview period for monitoring one of its acquisitions’ infrastructure before it was incorporated into their company tree as a subsidiary, and therefore affected the acquiring company’s rating. The PRB agreed that immediately after an acquisition, the security posture of the acquired company is unlikely to reflect the security posture of the acquirer; it takes time for the acquirer to make meaningful changes. To help establish an appropriate time period, the PRB solicited input from companies of various sizes on their typical acquisition timeline. Based on these discussions, the PRB decided to change the policy on acquisitions to the following: Bitsight will incorporate an acquired company into the acquirer’s rating six months after the transaction was closed, or three months after Bitsight finds out about the acquisition, whichever date is further in the future. Furthermore, acquirers who are Bitsight customers will be permitted a Security Performance Management (SPM) view into the acquired company’s infrastructure during the preview period. A future product enhancement will annotate the acquisition on both the acquirer’s and acquired company’s reports.

A rated organization requested a “refresh” of all of the findings in the Bitsight rating (which is not uncommon, and is supported by our policies). In the course of performing the refresh, Bitsight discovered a number of new assets which were not previously included on the report. The rated organization had been working through a strategic plan to remediate all of the negative findings on its assets, and the new assets were not accounted for in this plan. The rated organization requested that the new assets not be included on its report until the conclusion of that plan. Upon review, while we were sympathetic to the difficulties with the plan, we concluded that omitting the new findings altogether would be at odds with the integrity of the rating system, and would also be a disservice to other companies monitoring this particular organization as a vendor. Instead, we offered to create a breakout entity to capture the new assets (under the same original parent organization) and another breakout entity containing just the original assets, so that the rated organization could continue to use that to track progress against its plan.

A challenge was made to request that we change our grading policy for short DKIM (Domain Keys Identified Mail) keys, on the grounds that shorter keys are commonly used and do not pose significant risks. Bitsight’s current policy is to grade keys shorter than 2048 bits negatively. A brief technical background: DKIM is an authentication mechanism intended to verify that email was sent from the domain where it claims to originate. DKIM relies on a cryptographic signature algorithm, RSA, which in turn uses a cryptographic key of variable length (512, 1024, 2048, and 4096 bits are common) and is the product of two large secret prime numbers. The security of the algorithm depends on the difficulty of factoring the key; longer keys require more computational effort to “break.” As computing power has increased, key lengths which were once considered secure have become feasible to factor. As of 2015, NIST recommends RSA keys of at least 2048 bits. We also found that nearly all major email service vendors support 2048-bit keys. For these reasons, we elected to maintain the existing policy.

Bitsight’s policy is to include an IP address in an organization’s network map if there is a DNS (Domain Name System) record that associates one of the organization’s domain names with that IP address. Several organizations made disputes along the following lines: the organization divested itself or relinquished control of an IP address, but failed to update the corresponding DNS entry. Later, a negative event occurred on the IP address, was included on the organization’s report, and affected its security rating. In each case, the organization felt that because it did not, in fact, use the IP address at the time of the incident, the event should not be attributed to it. After review, we agreed with this position, but also concluded that the stale DNS record itself provided evidence of gaps within the organization’s security management practices. Our policy for these situations therefore changed as follows: upon receipt of sufficient evidence that the IP address changed hands, we will remove negative events, but we will also record a DNS hygiene security incident on the organization’s report.

An organization petitioned to change our grading policy for expired SSL/TLS certificates. Under current policy, expired certificates negatively affect an organization’s security rating. The organization argued that they should not, on the grounds that they don’t pose a security risk per se. We carefully considered the matter and reviewed recent recommendations for best practices from standards bodies such as NIST. This confirmed our understanding that continuing to use expired certificates reflects gaps in security hygiene, and may in fact pose some risks. We elected to maintain our current policy.

An organization experienced a data breach at one of its subsidiaries. The organization challenged our breach rating model, which would have reduced the rating of the entire company by the same amount as the (relatively small) subsidiary. In response, we studied the frequency of breaches vs. organization size, and found that empirically, reported breach frequency increases logarithmically with organization size, and therefore, the same breach carries more information about security performance when it occurs at a smaller organization vs. a larger one. After careful review, we updated our breach rating model to incorporate organization size, and we released the update in October 2020.

A rated organization challenged the inclusion of certain malware events in its corporate rating. These events were claimed to have originated from the company’s guest network. Our policy for guest networks on separate IP ranges is to allow the organization to create a breakout entity (under the same overall parent organization) to segregate these. In this case, however, the guest network shared the same egress IP address as the corporate network, so it was not technically feasible to determine, from an external perspective, where the events originated. Because it would be exceedingly difficult to create a uniform solution to handle this in a way applied fairly to all organizations we rate, we elected not to make a policy change at this time, but will continue to investigate technical solutions in the future.

To protect confidentiality and to respect responsible disclosure considerations, Bitsight’s terms of service prohibit rated entities from publicly sharing ratings, except that an organization can share its own rating (for marketing or any other purpose). However, one of our clients, which had the highest rating in its peer group, and one of the highest in its industry, wanted to use this fact to aid in its sales effort and share competitor ratings. After review, we decided that sharing comparisons to industry averages would not jeopardize confidentiality, so we decided to amend the terms of service to allow this. Furthermore, on a case-by-case basis, we may allow sharing of comparisons against a company’s peer group (if the group is sufficiently large to preserve anonymity of the companies within it).

Many companies conduct tests of their security controls (e.g. firewalls and intrusion detection systems) by running actual malware within controlled “sandbox” environments. Sandbox solutions are available from a number of well-established vendors, and running malware within a properly configured sandbox is expected to pose minimal security risk. The traffic generated by the malware running in a sandbox is very similar (or identical) to that generated by uncontrolled malware infections, and so is useful for ensuring that security controls are able to detect and/or block such traffic. However, this also makes the testing traffic externally indistinguishable from actual infections.

In the interest of consistency and scalability, Bitsight’s previous policy was to uniformly treat all apparent malware events identically, regardless of attestations that the events were due to deliberate tests. In response to a number of disputes, and the increasing prevalence of the practice of sandbox testing, however, we reexamined and modified this policy. We will now remove events that can be convincingly demonstrated to have originated from controlled tests; we will accept evidence such as screenshots of or logs from the malware testing software.