OVERVIEW
In December of 2019, the details of a critical vulnerability affecting certain versions of Citrix Application Delivery Controller (formerly known as NetScaler ADC) and Citrix Gateway servers were publicly disclosed.

The Proficio Threat Intelligence Team posted information about the vulnerability and its exploits in our Twitter Feed and issued a security advisory to our clients. In this blog, we share some of the findings from our own deep-dive investigations into the attack activities that we have observed in the wild as well as information that we have previously included within our advisory.

VULNERABILITY DETAILS
Citrix’s disclosed a significant amount of information regarding the CVE-2019-19781 vulnerability and exploit on publicly accessible channels. The details provided in their public release makes it easy for any potential attackers to recreate the exploit discovered.

The vulnerability is centered around a vulnerable parameter that allows for directory traversals due to the improper handling of the pathname. Attackers can exploit this vulnerability through crafted directory traversal requests to access sensitive files, create crafted XML files in the vulnerable server, and execute malicious code within those XML files without any authentication, effectively allowing for remote code execution.

This vulnerability is particularly serious because it allows an attacker to effectively obtain remote code execution capabilities on vulnerable devices. The exploit does not require access or knowledge regarding any user accounts and can be performed by any attacker. This makes this attack suitable for automation and mass-scanning, and we have indeed observed increased volumes of such automated attempts and attacks.

There have also been reports of attackers leveraging the exploit of the vulnerability to install malware and backdoors on vulnerable systems, preventing other attackers from exploiting and gaining access to the same system.

DETECTION AND DISCOVERY EFFORTS
Proficio’s Threat Intelligence Team collected a significant number of different IOCs and IOAs to identify potential exploit attempts. The IOCs and IOAs also include malware activity associated with successful exploit attempts against CVE-2019-19781.

We used reverse-engineering and contextualization techniques to research this vulnerability. By removing non-viable indicators that cannot be used for reliable detection or discovery, we were able to isolate high quality, useful and reliable threat indicators. This is particularly important given the limited visibility allowed for an MDRP/MSSP like Proficio.

Our detection and discovery efforts allowed us to identify a significant number of potentially successful exploit attempts against the vulnerable Citrix systems. This in turn allowed us to drive additional data collection efforts that served as a starting point for more deep-dive investigations for every potentially successful exploit attempt. 

NOTABLE CONCLUSIONS

1. Patching CVE-2019-19781 does not remove malicious artefacts left by successful exploits.

We have encountered several situations in which we were able to identify malicious post-exploit activity after successful exploit attempts against CVE-2019-19781, even though the vulnerability appears to have been patched. Identifying such activity after the patch is not necessarily an indication that the patch has failed, but a possible indication that malicious artefacts were already left behind by a successful exploit attempt prior to the patch being applied.

Organizations should note that compromised systems cannot be remediated by applying patches that were released to fix the vulnerability. Organizations will need to assess vulnerable systems to identify if any malicious artefacts remain from successful exploit attempts. Rebuilding the exploited system after the patch may be the only way to conclusively remove malicious artefacts from affected systems.

2. Not all successful exploit attempts were accompanied by post-exploit activity.

The exploits against CVE-2019-19781 are particularly suited for automation, and we have observed a significant increase in automated attacks and mass-scanning activities. Our investigation efforts have shown that in some cases, while we were able to identify that a successful exploit attempt is likely to have taken place, we were not able to identify any kind of post-exploit or suspicious activity from the vulnerable system that could have indicated live threat actor activity beyond automated scanning. Of course, this is no reason for complacency. Vulnerable systems should be patched and assessed as quickly as possible, especially if a successful exploit attempt against CVE-2019-19781 was observed. This assessment is required to positively identify the presence of any malicious artefacts on the vulnerable system. Should any be found, rebuilding the vulnerable system may be the only way to quickly and completely remove them from the affected system.

3. Security devices at the perimeter are the most useful log source for identifying and investigating successful exploits against CVE-2019-19781.

Having identified and investigated more than a dozen successful exploit attempts against CVE-2019-19781, it is interesting to note that the most useful logs for the investigations came from classic security devices like next-generation firewalls (NGFW) and intrusion detection/prevention systems (IDPS). Logs from such devices played a key role in all our deep-dive investigations and in some cases, happened to also be the only log sources with the requisite visibility for detection, discovery and investigation.

It is also interesting to note that logs from Citrix Netscalers were not particularly useful for the detection, discovery and investigation of exploit attempts against CVE-2019-19781. Of all the incidents in which we were able to positively identify the successful exploit attempts against vulnerable Citrix devices, we only made use of Citrix Netscaler logs in 7.6% of our investigations. Most of the logs that we were working with came from NGFWs and IDPS devices. Organizations affected by CVE-2019-19781 should review their logging configurations to ensure that the log events generated by their devices can be used for detection and discovery efforts. The last thing an organization wants is to realize that their current logging is not usable for the detection and discovery when there is a critical need to do so.

4. Not all IOCs are created equal – some IOCs are more useful than others.

We made use of a wide range of different IOCs that went through our qualification process. While we are confident in our selection of relevant IOCs, they did not play equal roles when it comes to our detection and discovery efforts. The following IOCs that have proven to be the most relevant and prevalent when it comes to performing deep-dive investigations:

• /ci.sh
  The filename references an installation script payload for a cryptocurrency miner. The payload creates a download loop for itself as a way to stage a backdoor for later while using cron jobs for persistence.
  Reference

• 95[.]179[.]163[.]186
  This IP address is known to used to download exploit payloads against CVE-2019-19781. One such payload would be the NOTROBIN malware. The IP address used to point towards the domain (vilarunners[dot]cat).
  Reference

• 185[.]178[.]45[.]221
  This IP address was used to host the file (ci.sh). Refer to the details on the IOC (ci.sh)

• 62[.]113[.]112[.]33
  This IP address was also used to host the file (ci.sh). Refer to the details on the IOC (ci.sh)

• 45[.]120[.]53[.]214
  Attackers are known to execute a curl command on successfully exploited systems in order to download a malicious shell script from this IP address onto a successfully exploited system.
  Reference

Even with the multitude of detection and discovery methodologies, the best way to deal with a known serious vulnerability is to patch the vulnerability when the patch becomes available. Since the initial disclosure of the vulnerability, Citrix has released patch updates for the impacted versions. After applying the patch fixes by Citrix, affected clients should also make use of the Verification Tool they provide to verify that the mitigation steps and patch fixes are applied correctly. Should the patch fixes provided by Citrix not be a suitable solution, the mitigation steps provided by Citrix alongside the vulnerability disclosure should be followed. Your cybersecurity team should closely follows vendor recommended best practices to ensure you’re patching known vulnerabilities as soon as possible.