Citrix NetScaler Gateway: NPS Extension for Azure MFA fails after introducing the Microsoft Domain Controller security baseline

Microsoft offers a nice set of security baseline GPOs, for direct use in your Active Directory environment. If you make use of the “MSFT Windows Server 2022 – Domain Controller” policy, your NPS installation might start to fail.

Introduction

Many of you probably have a Citrix NetScaler Gateway installation based on the following concept:

Manuel Winkel (deyda):

Microsoft Azure MFA Cloud Service in Citrix ADC

Thomas Preischl:

Citrix ADC / Netscaler Azure MFA Authentication

Those articles describe, how someone can implement Azure MFA with Microsoft Authenticator App pushOTP and an on-premises Microsoft NPS server, without making use of SAML, which is necessary, if you use the ICAProxy Gateway only license for Citrix NetScaler Gateway.

The Error

While implementing the Microsoft security baselines at a customers’ site, we also introduced the Domain Controllers security baseline called:
MSFT Windows Server 2022 – Domain Controller

MSFT Windows Server 2022 - Domain Controller
MSFT Windows Server 2022 – Domain Controller

A few minutes later, the external Citrix NetScaler Gateway authentication stopped working. Users would only receive the error:
Unknown username or password

Citrix NetScaler Gateway - Login failed
Citrix NetScaler Gateway – Login failed

Troubleshooting on the Citrix NetScaler Gateway through the aaad.debug showed the following error:

MS-CHAP error ERROR_AUTHENTICATION_FAILURE (691)

MS-CHAP error ERROR_AUTHENTICATION_FAILURE (691)
MS-CHAP error ERROR_AUTHENTICATION_FAILURE (691)

While the event viewer on the NPS server told us:

NPS Extension for Azure MFA: NPS Extension for Azure MFA only performs Secondary Auth for Radius requests in AccessAccept State. Request received for User Marco.Hofmann with response state AccessReject, ignoring request.

NPS Extension for Azure MFA only performs Secondary Auth for Radius requests in AccessAccept State. Request received for User Marco.Hofmann with response state AccessReject, ignoring request.
NPS Extension for Azure MFA only performs Secondary Auth for Radius requests in AccessAccept State. Request received for User Marco.Hofmann with response state AccessReject, ignoring request.

The solution

The error weren’t helpful at all. We spent quite too much time searching for an indicator of what might be the issue. We knew it must have to do with the Domain Controller security baseline. Finally, we stumbled at the following post from serverfault.com:

https://serverfault.com/questions/608227/authentication-via-radius-mschapv2-error-691

One of my colleagues was at a Microsoft conference having various discussions when it dawned on him that MSCHAPv2 relies on NTLM to generate the password challenges and responses. Now plain old MSCHAP and MSCHAPv2 (i.e. not EAP-MSCHAPv2 or PEAP) when used in Windows RAS services will use NTLMv1 by default.

As many of of you have already started to catch on, we, like many administrators, have disabled NTLMv1 on our DCs and as such the DCs will only accept NTLMv2 requests. This explains why the failure I continued to get was a “bad password” error. The password being sent to the DCs was in NTLMv1 format and was getting ignored.

This post then lets to a Microsoft article, with the solution:

https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/rras-vpn-connections-fail-ms-chapv2-authentication

For example, when you set this value to 5 (Send NTLMv2 response only. Refuse LM & NTLM), the DC won’t accept any requests that use NTLM authentication. When MS-CHAP or MS-CHAPv2 are configured, RAS in Windows Server 2008 R2 will default to NTLM to hash the password. Because the DC only accepts NTLMv2, the request will be denied.

Microsoft NPS on Windows Server 2019 and 2022 will use NTLMv1 by default, if you make use of MS-CHAPv2. And the MSFT Windows Server 2022 – Domain Controller security baseline will turn off NTLMv1 for your Domain Controllers, which will break your NPS server.

To enable NTLMv2 for MS-CHAPv2, you must set the following registry key on your NPS server, and restart the NPS service, and it will start working again:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RemoteAccess\Policy]
"Enable NTLMv2 Compatibility"=dword:00000001

Enable NTLMv2 Compatibility
Enable NTLMv2 Compatibility

I hope this will help someone else!

Potentially ongoing worldwide UDP:443 (EDT) DDoS amplify attack against Citrix (NetScaler) Gateway

Since 19 December 2020 7pm CET we see a possible worldwide DDOS amplify attack against Citrix Gateway UDP:443 DTLS EDT services.

Changelog

  • 11.01.2021: Added information about the new Citrix ADC Gateway (formerly NetScaler) firmware releases, which solve the memory leak issue with -helloVerifiyRequest
  • 24.12.2020: Added information about the official Citrix Knowledge Center article CTX289674
    Added a final summary, that repeats all possible solutions
    Maked it a lot clearer, that -helloVerifiyRequest doesn’t seem to work well
  • 22.12.2020: Added a warning note, that -helloVerifiyRequest doesn’t work on all Citrix ADC (NetScaler) firmware versions
  • 21.12.2020: Added a third possible solution regarding -helloVerifiyRequest
  • 21.12.2020: Initial version

The situation

During the night from Saturday (19.12.2020) to Sunday (20.12.2020) our Zabbix Monitoring informed us, that several Citrix Gateway VPX (50) appliances were at its license cap. We investigated the situation and soon found out, that we had 0 ICA sessions on most of them, hence no explanation for the traffic.

Zabbix Citrix Gateway Throughput Monitoring Graph
Zabbix Citrix Gateway Throughput Monitoring Graph

Continue reading “Potentially ongoing worldwide UDP:443 (EDT) DDoS amplify attack against Citrix (NetScaler) Gateway”

HowTo: Create a NetScaler Load Balancing vServer for Citrix Workspace Environment Management on the CLI

Since Citrix has released Workspace Environment Management 4.2 there is now a complete section about #WEM in the Citrix eDocs. Part of the new official documentation is a section about load balancing advices. But before that, there has already been a superior blog article about that topic by Ryan Revord.
But what is still missing is a complete overview of the necessary NetScaler CLI commands. That way you can import your WEM load balancing configuration in less than a minute! As I try to do everything in NetScaler on the CLI, I documented the steps during my first WEM deployment and wanted to share them with the community. Continue reading “HowTo: Create a NetScaler Load Balancing vServer for Citrix Workspace Environment Management on the CLI”

Citrix Certified Associate & Professional – Networking (CCA-N & CCP-N)

Today I passed exam 1Y0-351 after I visited the early access classroom:
CNS-222EAI Early Access: NetScaler for Apps and Desktops
in may 2016. I already passed the CCA-N in August 2016, but forgot to post it here, if I remember correctly.

Citrix Certified Associate - Networking (CCA - N) Citrix Certified Professional - Networking (CCP - N)

I’m would like to say, that the new NetScaler Gateway classroom course CNS-222 was a great help and the instructor Paul Berr did a great job!

Since the course I barely use the WebGui anymore, and achieved great progress in CLI only configurations. I built my self several CLI templates for regular tasks and improved my skills from deployment to deployment.

You can see my earned titles here:
https://www.youracclaim.com/users/marco-hofmann

NetScaler CLI Syntax highlighting with Notepad++

Last Friday I had to write a large NetScaler config file. Most of the time I do this in a plain text editor. Then I asked myself, if there is any form of code editor for NetScaler. I didn’t found one after a quick research, but I found a syntax highlighting definition for Notepad++ which is a lot better than nothing 😀
Here is the source, it’s a German website, but you can’t overlook the following line:

“Hier die aktuelle Version [Stand 29.02.2016] : Netscaler.xml”
which means:
“Here is the latest version [Last Update 29th February 2016] : Netscaler.xml”

R33NET BLOG: NetScaler Syntax highlighting for Notepad++