Introduction
Earlier this summer, the internet was taken by storm by three critical vulnerabilities affecting MOVEit Transfer, a file-transferring software solution widely used by high-profile companies to store and share, in some cases, even the most sensitive business information.
In a previous post, we’ve already detailed some insights on the evolution of affected servers on the internet during the timeline of the incidents and vulnerability publications. However, in this post, we aim to showcase, on a technical level, the methodology discovered and used by the Bitsight Security Research team that allowed us to pinpoint not only servers running the software, but their versions, and with this, a high degree of confidence of exposure to the critical CVEs.
The setup
The initial phase of any research of this kind involves setting up a testing environment, where we replicate an exact match of typical production installation of the software we’re researching, both vulnerable and not vulnerable versions of it. This will allow us to explore all the available functionality of the product, test any payloads without impacting real-world instances, and look for clues or different behaviors between the patched and unpatched instances.
Detecting the MOVEit Transfer Product
In the first few hours after the publication of the CVE-2023-34362 Advisory (2 June 2023), several cybersecurity news outlets and Twitter accounts started to report some Indicators of Compromise (IoCs) of exploitation of this CVE in the wild, now attributed to the Cl0p Ransomware gang.
Along with these IoCs, some queries for internet engines were shared, such as the http.favicon.hash:989289239 query for Shodan, which worked by querying which internet servers were using the Favicon associated with the MOVEit company (and thus the MOVEit Transfer product).
https://twitter.com/HunterMapping/status/1664752220384968704
Using this, multiple sources claimed that there were around 2,500 installations of MOVEit Transfer online and reachable at the time of the release of the advisory.
Our research concluded, however, that these numbers were inflated by three factors:
- Historical observations were included in the numbers, so not all instances were still online after the advisory publishing.
- Results could appear on multiple ports but were not being deduplicated by IP address (some IPs were serving the software both over HTTP and HTTPS).
- This Favicon Hash also erroneously identified other products from Progress (the same vendor and thus the same icon) - such as MOVEit Automation.
These adjustments were considered in our initial scans, and at the time we estimated the total number of MOVEit Transfer instances live at the time was closer to 1,700.
By leveraging this technique, applying our analysis to reduce false positives, and loading it to our internet-scanning infrastructure, we were able to provide our customers with a first round of preliminary exposure results within 48 hours.
Now the hard part: the version
As soon as the first technical details of the vulnerability were published, we quickly came to the conclusion that it would be extremely unlikely we would want to use any kind of active probe. An active probe issues requests within the affected functionality to confirm if the software has been patched.
Due to the nature of the vulnerability itself, an SQL Injection, poking anywhere near the injection points (which, keep in mind, were still not public at this time) would be considered an intrusive, potentially dangerous technique, especially when loaded into our internet-wide scanning infrastructure.
With this being the case, after implementing the product detection capability described above, we started channeling our efforts into ways of figuring out the version of the running software instance. If we know the version, we can compare it against the versions in the vulnerability’s advisory to infer exposure with high confidence.
In some software, a version is trivial information to retrieve, as it’s provided, for example, in a banner underneath the login interface, in a certain unauthenticated /version endpoint, etc. However, for MOVEit Transfer, this was not the case.
The first breakthrough we reached was using one of the main endpoints in the application, /human.aspx?arg12=infotech. This was one of the application’s main settings pages, which we discovered right away after setting up our test environment. When authenticated, an administrator could see the installed version of the software in this endpoint. Unfortunately, this is not useful for us, because our internet-wide scan won’t be able to authenticate to the MOVEit Application.
This /human.aspx?arg12=infotech page was also served without authentication, albeit with much less detail. Unfortunately, the unauthenticated page is displayed without the full version that an authenticated administrator could see.
However, even in the unauthenticated version of the page, a breadcrumb was left that allowed us to detect at least the major version of the running software.
On the page, there was a dynamic link to the documentation that changed according to the major version installed in the server, according to a known mapping. For example:
If the server links to:
docs.ipswitch.com/MOVEit/Transfer2023/Help/- it’s running a 15.X versiondocs.ipswitch.com/MOVEit/Transfer2022/Help/- it’s running a 14.X version- Etc.
Since Progress initially stated that, for major versions 12 and below, there wouldn’t be a release of the fix and that an upgrade to a supported version was needed, we could use this to at least find servers running an unsupported version, and flag those as vulnerable.
However, this was insufficient, as our first scans with this detection revealed that almost all servers on the internet running MOVEit Transfer were not using versions below 12 anyway. We still needed to find a way to retrieve the full version for all cases to be able to provide any value.
This led us to our final research breakthrough.
Shipped along with the installation of the MOVEit Transfer server software is a small utility tool in the form of an executable called MOVEit Transfer Checker. When researching, we discovered that this was a utility for troubleshooting connectivity issues between components of the MOVEit Transfer ecosystem. It tested sample file uploads to the server through the Web API, SSH, and FTP, as well as tested the SMTP connection needed to send email notifications of the service.
When testing this utility, and looking at its debug logs, we could see that, at the beginning of the test flow, the utility app retrieved and logged the full version of the instance before moving forward with the test upload via Web API. Bingo!
Using Wireshark, we intercepted the first part of the test flow and managed to obtain the exact HTTP request made by the utility to obtain the full version information, as shown below:
The utility performs a POST request to /MOVEitisapi/MOVEitisapi.dll?action=capa (“capa” which we assume to be short for “capabilities” similar to the IMAP/POP command), which returns a x-MOVEitisapi-version header containing the full version, including the minor and patch numbers, required to confirm if the CVE patch has already been applied.
Finally, we replicated this request against our test instance before and after applying the patches for the newer CVEs and confirmed the version reported remained accurate, as seen below.
After all this, it was a matter of polishing up the request, optimizing it to remove anything that wasn’t necessary to obtain the version information (while doing this, we realized it was also possible to retrieve it using a GET request to the same endpoint) and adding it to our internet-scanning engine.
TLDR: The final detection workflow
The fingerprinting method for MOVEit Transfer software is divided into three phases.
A scan is performed against every IPv4 address on the internet (bar some exceptions - blacklists, etc.), on the ports which are usually associated with default MOVEit Transfer configurations (based on their documentation), and all the ports indexed to MOVEit Transfer software on Internet Search Engines. (80, 8080, 8090, 443, 8443, 9443).
Then, when an ip:port pair is live we:
- Confirm if it’s a MOVEit Transfer by analyzing the favicon hash and cleaning up false-positives
- Replicate the request done by the MOVEit Transfer Check utility tool to obtain the
x-MOVEitisapi-versionheader with the full version - In case it fails, we attempt to obtain just the major version by parsing the release year from the dynamic link to the documentation in the unauthenticated settings page
After this fingerprinting process, we compare the versions we have obtained against the versions in the CVE Advisories to infer if the server is affected or not.
Concluding remarks
The MOVEit Transfer Security Event showed the importance, capabilities, and speed of the Bitsight Vulnerability Research team. Our dedicated team enabled us to formulate strong custom-built detections that were not matched by any competitors at the time, thus offering unique visibility into the exposure landscape of our customers - within just a few days of the publication of the critical vulnerabilities’ advisory.