Potentially ongoing worldwide UDP:443 (EDT) DDoS amplify attack against Citrix (NetScaler) Gateway

Since 19 December 2020 7pm CET we see a possible worldwide DDOS amplify attack against Citrix Gateway UDP:443 DTLS EDT services.

Zabbix Citrix Gateway Throughput Monitoring Graph

Changelog

  • 11.01.2021: Added information about the new Citrix ADC Gateway (formerly NetScaler) firmware releases, which solve the memory leak issue with -helloVerifiyRequest
  • 24.12.2020: Added information about the official Citrix Knowledge Center article CTX289674
    Added a final summary, that repeats all possible solutions
    Maked it a lot clearer, that -helloVerifiyRequest doesn’t seem to work well
  • 22.12.2020: Added a warning note, that -helloVerifiyRequest doesn’t work on all Citrix ADC (NetScaler) firmware versions
  • 21.12.2020: Added a third possible solution regarding -helloVerifiyRequest
  • 21.12.2020: Initial version

The situation

During the night from Saturday (19.12.2020) to Sunday (20.12.2020) our Zabbix Monitoring informed us, that several Citrix Gateway VPX (50) appliances were at its license cap. We investigated the situation and soon found out, that we had 0 ICA sessions on most of them, hence no explanation for the traffic.

Zabbix Citrix Gateway Throughput Monitoring Graph
Zabbix Citrix Gateway Throughput Monitoring Graph

The search

Before we started the investigation, we took each and every Citrix Gateway VPX offline, to gather time for the research.
We considered several possible root causes for the issue, for example:

  • Malware on the VPX appliance
    (although all are up-to-date, but you may never know)
  • VPX being part of a botnet
  • Malware inside the customers network, leveraging the VPX in some form or another
  • Bruteforce attacks, although all appliances have 2FA or MFA configured
  • DDoS

After quite some tests we were quite sure, that only the last option would be viable:

Packet capture from the corporate firewall
Packet capture from the corporate firewall

To proof our theories, we brought test appliances back online and only blocked the potential source IPs of the DDoS at the corporate firewall level. The bandwidth consumption on the Citrix VPX immediately went down to zero.

One possible solution

We then started blocking the source IPs one by one on the corporate firewall, but soon realized, that this would be an impossible task, although we saw immediate relief. So we decided to completely block UDP:443 targeting the customers Gateway VIP on the corporate firewall level, as we are luckily not dependent on Citrix EDT.

The next day

During Monday morning (21.12.2020) we found additional evidence in the EUC community:

World of EUC Slack
World of EUC Slack

And on Twitter, that we were not alone, and that our conclusions were somewhat correct:

We then double checked all our managed corporate Firewalls, if we correctly blocked UDP:443 DTLS for our Citrix Gateway customers.

Another possible solution

Additionally we had one case, where we weren’t able to block UDP:443 on the corporate firewall, so we disabled DTLS on the Citrix Gateway virtual server instead.

Gateway virtual server DTLS
Gateway virtual server DTLS

Third possible solution

This solution is only recommended, if you are on firmware version:

  • 13.0-71.44 and later
  • 12.1-60.19 and later
  • 11.1-65.16 and later

Otherwise, you will have memory leaks which will crash your Citrix ADC, see paragraph below!

By now several Citrix ADC experts did came up with a third solution, which does not require to prevent any EDT/UDP DTLS connection on port 443. With the following command, you can enable that those requests are properly validated by Citrix ADC. They also mention, that this feature comes with a small performance degradation on the initial connection. On the other hand you might not need to alter your corporate firewall rules, and are still able to benefit from EDT protocol:

set ssl dtlsProfile nsdtls_default_profile -helloVerifyRequest ENABLED

Update regarding -helloVerifiyRequest

Update: This issue has been solved, if you are on firmware version:

  • 13.0-71.44 and later
  • 12.1-60.19 and later
  • 11.1-65.16 and later

There are confirmed cases on Twitter and in my blog comments (see below), that the -helloVerifiyRequest option doesn’t work reliable on all different Citrix ADC (NetScaler) firmware versions out there. Please test this setting properly in your environment. I don’t have an exact list, but there are several firmware versions out there, where -helloVerifiyRequest will lead to a memory leak, which will potentially crash you Citrix ADC after a few hours.

Zabbix Monitoring

After the dust was settled, I also added an additional Trigger to our MSP Zabbix monitoring, were we compare now, if the traffic on the VPX is unusual high, in comparison to the number of active ICA sessions.
It’s very simple, but gets the job done:

{Template SNMP Citrix Gateway:Throughput.avg(15m)} > {Template SNMP Citrix Gateway:aaaCurICAOnlyConn.avg(15m)}

Citrix Knowledge Center Article

In the meantime (24.12.2020 here in Europe) Citrix has published an official statement regarding this subject in their Knowledge Center:

https://support.citrix.com/article/CTX289674

The official statement from Citrix is, to disable the DTLS feature on the Citrix Gateway virtual server.
Additionally Citrix announces an upcoming enhancement for the Citrix ADC (NetScaler) Firmware, where this special attack will be eliminated.
This is to be expected 12.01.2021.

Update 11.02.2021: Citrix has released new firmware versions. These firmware fixes the problems with -helloVerifiyRequest and makes it a working solution. With the current firmware and -helloVerifiyRequest enabled, you are safe from the DDoS amplify attack, while still being able to offer the Citrix EDT protocol.

Summary

There are three possible solutions, I list those in my personal preferred order:

  1. Set -helloVerifyRequest ENABLED in the DTLS profile.
    It has been reported by many comments here on my blog and on Twitter, that there are Citrix ADC firmware versions, which will have a memory leak and crash a few hours later. This was fixed by Citrix, please check: CTX289674. Fixed in:

    1. 13.0-71.44 and later
    2. 12.1-60.19 and later
    3. 11.1-65.16 and later
  2. Block UDP:443 traffic targeting the Citrix Gateway VIP at the corporate firewall level, to prevent your states table being overwhelmed.
  3. Disable the DTLS feature on the Citrix Gateway virtual server, as recommended by Citrix, if you are not ready for the new firmware version, yet.

Final words

To find out, if you are under attack, take a look at Citrix ADC Dashboard and look at the current Throughput and ask yourself if this makes sense:

Citrix ADC Throughput
Citrix ADC Throughput

Additionally you might want to tcpdump and/or Wireshark you corporate Firewall WAN interface and search for UDP:443 traffic pointing at your Citrix Gateway VIP, if it looks somewhat similar to my Wireshark screenshot above.

Also subscribe to the on-going discussion in the World of EUC Slack, which is a tremendous source for everything going on in the EUC space.

Author: Marco

Marco is an IT-System administrator and IT-Consultant with 10+ years experience. He is specialized in the delivery of virtual Apps and Desktops with Citrix solutions. In 2017 he has been awarded Citrix Technology Advocate by Citrix for his community work (#CTA). His second core area is availability & performance monitoring with Zabbix, a leading open-source solution. His employer is the German IT-Company ANAXCO, which is developing a Transport Management Software (TMS) based on Microsoft Dynamics AX. More about Marco

134 thoughts on “Potentially ongoing worldwide UDP:443 (EDT) DDoS amplify attack against Citrix (NetScaler) Gateway”

  1. We have been seeing this for the last 48 hours against multiple netscalers also.

  2. The 3rd solution doesn’t work. Memory is keep increasing and finally the whole netscaler die. Finally i use the first solution and now is monitoring.

    1. I read several possible comments on Twitter, about the third solution being a good solution, but blocking the traffic at the corporate firewall level is still a valid and reliable option! Good you got it solved for now!

      1. Hi Marco,
        Thanks for sharing these details. Enabling -helloVerifyRequest causes NetScaler NS13.0: Build 61.48.nc to become unresponsive. Not recommended. Best to disable DTLS entirely until further notice.
        Best regards,

  3. Hi,
    Good job.
    What will be the impact to SSL VPN user service by disabling UDP/443 at the edge?
    finding conflicting info from citrix about this.
    thanks

    1. To be honest, I don’t know, as we don’t make use of the SSL VPN feature. We have all our Citrix Gateways in ICA proxy mode.

  4. I would either disable DTLS or block with firewall. The “set ssl dtlsProfile nsdtls_default_profile -helloVerifyRequest ENABLED” causes a memory leak (validated in v11.1 and v12.1) and you will lose access to your ADCs.

    1. Yes, I know. I already update the blog post with that new information, did you see that? Do you think I should make this more clear?
      At our company we didn’t use that method, but I added it, as many people wanted to know it.
      We simply blocked UDP 443 at the corporate firewall since Sunday and are happy at the moment.

  5. We have identified the same issue in my compagny. After a exchange with Citrix Support a public post must arrived. The Citrix response could be to use the option “Hello verify” or disable DTLS if you not use EDT but they don’t say if there is an issue with a memory leak it’s depend of the Netscaler build…

  6. We had same issue.

    on our ELK I registered more than 3400 unique external IP adresses that are participating in the attack.
    So blocking the IPs one by one not the berst way to prevent the DDOS.

  7. I manage to mitigate this by learning the attack layout, you will see the payload of the packets and its all the same they are just using blocks of ip addresses to send small amout of traffic, they to capture the traffic and read the packet data. You will see the similarities of every single packets.

    1. Thank you very much. I will update the blog post accordingly.
      For us, about 90% of our public Citrix Gateway devices were under attack.
      I also received some comments about Citrix ADC (NetScaler) Admins who weren’t affected and asked me for traces.
      Also, maybe not everyone was DTLS enabled or makes use of EDT.

      1. Hello Marco,

        Does this threat affects the NS VPN which are exposed to internet?
        What is we have VPN gateway with Site-Site corporate access?

    2. I agree, Stephen. I believe all Citrix Gateways with EDT enabled are affected. Since there are lists of Gateways available online, the statement by Citrix is misleading at best. Sure, it is true that not all Gateways have EDT enabled and udp port 443 exposed. Additionally I’ve found the attacks are intermittent and not all environments are sufficiently impacted by the traffic generated for the attack to be detected and attributed to this vulnerability. In any case it’s good news that an update is coming!

      1. I do not believe by default DTLS is enabled (unless some version/builds set that by default now). So unless you are trying to utilize EDT for ICA and DTLS for full VPN, the 443/UDP would not be listening. There is a third option which would be someone turning the DTLS knob to enabled without understanding why they did that.. Maybe the enhancement the CTX article is talking about is fixing the memory leak in “-helloVerifyRequest ENABLED”.

        Has anyone been able to utilize the “hello verify” without losing NSIP access to their MPX/VPX instances after a few hours? The memory leak time to loss of NSIP access may vary between version/build and the amount of malicious traffic.

  8. My NSVPN is doing tons of pings to the following IPs. DTLS was disabled and the device rebooted but to no avail. I wonder if my NS has been compromised, anyone else seeing this?

    193.231.169.133
    193.231.169.155
    193.231.169.199
    193.231.169.144

  9. What OID(s) are you using in your Zabbix template to get the ‘Template SNMP Citrix Gateway:Throughput’ value? I’ve got the aaaCurICAOnlyConn OID from the Citrix docs, but can’t work out how you’ve pulled the single Throughput value.

    1. Coo, let’s find out! I’ll be pushing the update for a few customers this evening.
      All my customers with DTLS enabled (whether manually enabled or not) were affected: udp/443 was blocked at the firewall to mitigate.

      1. No issues to report with the updated 12.1 and 13.0 releases after 24 hrs. Enabling helloVerifyRequest would’ve hung up the NetScaler way earlier on previous releases.
        DTLS/EDT humming along nicely once again. Will be scheduling further upgrades.

  10. Hi Marco,
    We are also using Zabbix to monitor our NS.
    Would you be able to share your template in regards to this “{Template SNMP Citrix Gateway:Throughput.avg(15m)} > {Template SNMP Citrix Gateway:aaaCurICAOnlyConn.avg(15m)}”.
    If the template is from an online source could you share?
    If none of those are possible could you share the names and OID’s of the values used in that trigger.
    Thank you sir.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.