My first time: Citrix Machine Creation Services (MCS)

After years of MCS virginity I decided it’s finally time to ditch the little farms and try out good ol’ Citrix Machine Creation Services.

Introduction

So today is my very first time: After years of MCS virginity I decided it’s finally time to ditch the little farms and try out good ol’ Citrix Machine Creation Services. In the last ten years I almost exclusively installed small deployments. The big ones have about 150 concurrent user. All are build upon XenApp 6.5 or XenApp 7.6+ with static persistent virtual machines. I always told myself, that static persistent virtual machines, together with a fully automated patch management (for example: PDQ) are enough. And this is still true, because the maintenance effort is virtually not existent. But it really bugs me, that I’m not equally familiar with at least one of the provisioning methods. You might ask, why I don’t try to learn PVS instead. Well, the simple reason is that my stomach tells me not to. The more valid reason is that MCS is included in every XenApp license and doesn’t require additional infrastructure. And additional infrastructure is always a really big topic for the customer.

This blog post won’t be a real classical Blog HowTo Guide, but more of a report of my journey to help me keep track about what I do. Maybe others suffer the same knowledge gap and are interested in my findings and the path I take.

Upsides and Downsides from a beginners perspective

Before this project, when I’ve had no real MCS experience I always thought about the following points, and even if they bring no real content to this blog post, I would still like to write them down before we start. You may skip this part if you like 😉

Upsides:

  • I could easily change the count of the XenApp VDA by adding more clones.
  • The cloned VMs would be 100% identical, compared to my static VMs I use today, which get unequal over time.
  • I get some kind of VM or generation versioning through snapshot management, if done correctly.
  • MCS RAM cache can help with write IO.

Downsides:

  • I have no real plan about how to handle Windows, Office and Application Updates. Since MCS cloned machines are read-only I think, each change I have now fully automated through PDQ Deploy would be lost after a weekly reboot for example.
  • Does this mean, instead of (nearly) no work today, I would have to roll-out new updated snapshots 2-4 times a month manually, to keep the same level I’m used to?
  • What about updates on weekdays? Every now and then customers decide to change things on weekdays. Or a new Java versions breaks a Browser Application and requires an immediate update. This will be a lot harder with non-persistent VMs + a catalog update always requires a reboot I think.

Initial Situation

My company’s test environment has the following pre-condition:

  • VMware vSphere 6.5
  • Two XenApp 7.15 Delivery Controller
  • Different static persistent Windows Server 2012 R2 & 2016 XenApp VDA
  • An up and running fully functional Active Directory Environment based on Windows Server 2016

Getting started

Create a Master Image

  1. Install a fresh Windows Server 2016 Datacenter virtual machine.
  2. Install Hypervisor Tools -> VMware Tools:
    setup64.exe /S /v"/qn REBOOT=R"
  3. Fully patch the OS (for example with ABC-Update and automated reboots):
    ABC-Update.exe /A:Install /S:MSUpdate /R:10 /Log_Append:c:\WSUS.log /Exit:Restart
  4. Install the Citrix XenApp Virtual Delivery Agent, for example:
    import-module servermanager
    Add-WindowsFeature Desktop-Experience

    Reboot

    import-module servermanager
    Add-WindowsFeature RDS-RD-Server

    Reboot

    VDAServerSetup_7.18.exe /quiet /components vda /controllers xa-wms2016-01.int.anaxco.de,xa-wms2016-02.int.anaxco.de /enable_hdx_ports /enable_hdx_udp_ports /enable_framehawk_port /enable_real_time_transport /enable_remote_assistance /virtualmachine /optimize /noreboot /logpath C:\
  5. Install the Citrix Workspace Environment Management Agent:
    Citrix Workspace Environment Management Agent v4.04.00.00 Setup.exe /s /v"/qn /norestart"
    C:\Windows\Microsoft.NET\Framework64\v4.0.30319\ngen.exe update
    C:\Windows\Microsoft.NET\Framework64\v4.0.30319\ngen.exe eqi 3
  6. Install the Citrix Connection Quality Indicator (CTX220774):
    msiexec.exe /i "CitrixCQI.msi" ALLUSERS=1 /qn /norestart /log output.log
  7. Install Base Image Script Framework (BIS-F) 6.1.0 -> Documentation
    setup.exe /SILENT

    1. Import and configure ADMX; example:

      BIS-F GPO
      BIS-F GPO
    2. Provide 3rd party dependencies; I decided for:
      1. Delprof2
      2. Citrix Optimizer
  8. Apply additional OS optimizations based on your personal style.
    An excellent source for for this topic is: Carl Stalhood
    I will name a few examples:

    1. Disable Hotplug of VMware VMXNET3
    2. Remove VMware Tray Icon
    3. Black Screen on Windows Server 2016 / CTX225819
    4. Set UseProfilePathExtensionVersion registry value to 1
    5. Server 2016 desktop icons flickering @ discussions.citrix.com
    6. Edit pagefile.sys size
    7. Check NTFS permissions on C:\
    8. etc.
  9. Install some software, for example:
    1. 7-Zip
    2. Adobe Acrobe Reader DC Classic
    3. ASG-Remote Desktop
    4. (ControlUp Agent)
    5. (ESET AntiVirus + Agent)
    6. ESTOS ProCall CTO Client
    7. (fslogix)
    8. Google Chrome
    9. Greenshot
    10. MailStore
    11. Microsoft Office 365 ProPlus
    12. Mozilla Firefox
    13. Notepad++
    14. PutTTY + KiTTY
    15. TeamViewer
    16. VMware Remote Console
    17. VMware vSphere Client
    18. WinSCP
    19. (Zabbix Agent)
  10. Get your Computer and User Group Policy Objects in place with everything you need, for example remove typical lock down policies from the Master during creation and later maintenance:

    GPO
    GPO
  11. Execute BIS-F to seal and shutdown your Master Image:
    Execute BIS-F
    Execute BIS-F

    Hint: I had several problems with that step because of very strict Anti-Virus policies. I had a great discussion at the BIS-F Slack with Matthias Schlimm about this problem. We narrowed it down to some AppLocker like behavior, but it turned out we had some ESET policy in place, which prevented child processes from PowerShell.

    Troubleshooting with Matthias Schlimm
    Troubleshooting with Matthias Schlimm

The Citrix Studio Hosting Connection

  1. Start Citrix Studio and create a new Hosting connection. I will to this example for a VMware vSphere 6.5 Cluster:

    Citrix Studio Hosting Connection
    Citrix Studio Hosting Connection
  2. Trust the self-signed certificate if necessary:

    Trust the self-signed certificate
    Trust the self-signed certificate
  3. Set up your storage as desired, consult the storage administrator if you need help. IMHO this decision is not a Citrix Administrators Task:

    Storage Management
    Storage Management
  4. Select storage LUNs as desired:
    Storage Selection
    Storage Selection

    Hint: As I had to learn the hard way, selecting all LUNs available wasn’t such a good idea and your storage administrator won’t be very pleased. As it turns out, MCS deploys a clone of your master image on every LUN selected. So in my case I deployed the VMDK to 17 LUNs while I only intended to get used to MCS with two VMs:

    A hint from Jarian Gibson
    A hint from Jarian Gibson

    Turns out MCS needs a read-only disk on every selected LUN, independent from the VM count. Plan and size accordingly:
    Source: Citrix Product Documentation – XenApp Current Release

    MCS creates a full copy of the snapshot and places the copy on each storage location defined in the host connection.

    In the last chapter of this blog post is a description about how I handled my error.

  5. Select the VLANs that will be available when you create the machine catalog. This settings won’t be used until the catalog will be created. For example:
    If you select three VLANs and plan to later provision VMs with one NIC, you have to select one of these three VLANs for the single NIC. The other two won’t be used. Consult the network administrator if you need help:

    Network
    Network
  6. Confirm the summary dialogue.

The MCS Wizard

  1. Login to your Hypervisor and create a snapshot manually. The VM should already be powered off, after you executed the BIS-F script. In my case this is the Flash Player based vSphere WebClient 6.5:

    Create a VMware Snapshot
    Create a VMware Snapshot
  2. Create a new Machine Catalog in Citrix Studio:

    Create Machine Catalog
    Create Machine Catalog
  3. Because I mainly work with XenApp / RDSH / Server OS or whatever you might call it these days, I select Server OS:

    Operating System
    Operating System
  4. We set up a VMware virtual machine as our master image, so we have a power managed virtual machine. And as this guide is about MCS, we won’t select PVS. Always keeping things simple:

    Machine Management - Power managed - Machine Creation Services
    Machine Management – Power managed – Machine Creation Services
  5. From the list of VMs we search our powered off master machine.
    Important: In step one we created a snapshot manually! You could select the virtual machine itself as a source, and this will work without any problems. But you will lose control over your image management! If you let Citrix Studio do the work for you, all the snapshots created will have the same name and description.
    Source: Citrix Product Documentation – XenApp Current Release

    If you selected a master image (rather than a snapshot), MCS creates a snapshot.

    If you do it manually (or maybe scripted sometime later?) you will have control over the name and the description, which brings you to some form of MCS golden master image versioning. Always keep track of those versions and the changes made, so you know exactly where to go, if you ever have to roll back a catalog!

    Select Golden Master Image Snapshot
    Select Golden Master Image Snapshot
  6. Select one of the VLANs you configured in the Hosting connection for the cloned virtual machines:

    Select Network
    Select Network
  7. The next page is very important, you have four values:
    Configure a cache for temporary data on each machine.
    Configure a cache for temporary data on each machine.
    1. Virtual machine count
      The machine count is self-explanatory, just set the number of XenApp workers you need and you’re set!
    2. Total memory for each machine
      This depends on your workload. There is no recommendation here other then: Test! Test! Test!
      Either do it manually, or use tools. For example, you could ask your monitoring administrator (#Zabbix) to provide you with hard facts about your workload. Or use some industrial standard tools like LoginVSI to simulate a real workload and get values for a proper sizing. But I can at least provide you with a link to an interesting blog post about sizing:
      Daniel Feller: Sizing Windows 2016, Windows 2012 and Windows 10 Virtual Machines
    3. A temporary RAM cache for writing operations
      With the release of XenApp 7.9 Citrix introduced a new feature for MCS, previously only know to PVS: A memory writing cache. This cache handles all writing operation prior to writing them to the temporary disk. This can deliver a serious IO reduction to your storage. The amount of RAM you configure here, gets cut from the “Total memory for each machine” you defined in step 2. So add more total RAM accordingly to the RAM cache disk size.
      You can find a lot more information about this topic in the links I collected in step 5.
    4. A temporary Disk as overflow for the RAM cache
      The memory RAM cache disk is limited (in most cases). So lets say you allocate about 2-6 GB memory cache. Depending on your workload, this could fill up sooner or later. With a small or no overflow disk you will get a Bluescreen #BSOD very fast. The OS tries to write, runs out of space and crashes. So your overflow disk is there to step in, if the RAM cache can’t handle the write requests anymore. So if you would like to use the RAM cache feature, never use it without an overflow disk. A typical suggestion is, that the disk should be at least the size of the free disk space of your VDA worker.
      You can find a lot more information about this topic in the links I collected in step 5.
    5. Additional Information about MCS storage optimization
      https://www.citrix.com/blogs/2016/08/03/introducing-mcs-storage-optimisation/
      https://www.controlup.com/blog/everything-you-need-to-know-about-the-new-citrix-mcs-io-acceleration/
      https://www.jgspiers.com/machine-creation-services-storage-ram-disk-cache/
  8. Now the MCS cloned machines need to have computer accounts in Active Directory. Here you can select, if new or existing accounts should be used. Additionally you choose the OU where the computer accounts will be created. Important here is the naming scheme. You can see my example choice in the screenshot:

    Active Directory Computer Accounts
    Active Directory Computer Accounts
  9. Finally you choose a name for the Machine Catalog. As soon as you click finish, MCS will start its work:

    Summary - Machine Catalog name
    Summary – Machine Catalog name
  10. MCS at work in Citrix Studio:

    MCS at work
    MCS at work
  11. Screenshots from VMware vSphere Client 6.5 (HTML5) during the Machine Catalog creation:
    VMware Machine creation progress
    VMware Machine creation progress

    VMware Machine creation progress
    VMware Machine creation progress
  12. After the process finishes, additional offline VMs will be available for use.

    New virtual machines in vSphere Web Client
    New virtual machines in vSphere Web Client
  13. The machines will stay offline, until Citrix Studio tells them to start up. For example, after they are assigned to a Delivery Group.

Delivery Group

  1. Now that we have machines available, it’s time to create a Delivery Group, so we can access our work from a users perspective for the first time. Select Create Delivery Group from within Citrix Studio:

    Create Delivery Group
    Create Delivery Group
  2. Select the previously created MCS catalog and enter the number of machines you want to use. Normally this is equal to the number of virtual machines you created:

    Select Machine Catalog
    Select Machine Catalog
  3. Select a User or a security group for the Delivery Group if desired:

    Select Users
    Select Users
  4. Publish different applications if desired:
    (I presume you know what a published application is, how and why to use it etc.)

    Create published applications
    Create published applications
  5. Create a published Desktop if desired:

    Create published Desktop
    Create published Desktop
  6. Give the Delivery Group an appropriate name:

    Summary - Delivery Group name
    Summary – Delivery Group name
  7. And done! By now your previously offline MCS VMs should boot up. After they ran through sysprep and BIS-F (check the log files in the UNC path you configured!), the Citrix VDA will register itself with the Delivery Controller(s), and your newly created Apps and Desktops will be available through Citrix Storefront!

Update Machine Catalog

After our first successful MCS roll-out, it’s time to talk about updates. I personally have no real practical experience with Machine Catalog Updates. My knowledge here is more theoretical, based on a lot of reading and many discussions on Slack and community meet-ups. So there are several challenges to solve in my opinion, just to list a few that come to my mind:

Con’s

  1. Compared to static persistent Server OS VDA RDSH VMs, all changes that happen during the VMs up-time will be “lost” after a reboot. That is because the cloned MCS disks are read only. So your classical Software-Deployment Auto Schedules would still work in some way, like a Java update for example, but as soon as you reboot, that will be reset.
  2. Without Microsoft System Center Configuration Manager (SCCM) no scheduled roll-out seems to be possible. This is a big disadvantage in my opinion. It seems like PowerShell would be the way to go. The Citrix PowerShell CMDlets seem to offer advanced features to control MCS catalog updates. If this could be solved, any Task Scheduler could solve the timing problem.
    Rollout Strategy
    Rollout Strategy

    EDIT: I got informed by Jarian Gibson, that the wording in the Rollout Strategy is not the best, or may just plain wrong:

    Troubleshooting with Jarian Gibson
    Troubleshooting with Jarian Gibson

    This seems to be true, so my initial description is wrong. As detailed in the Citrix Discussions post, you can schedule a rollout without SCCM. The trick is that the next VDA reboot has to be initiated through Citrix Studio. This can also be achieved through a scheduled reboot inside the Delivery Group. More on this later.
    Source: https://discussions.citrix.com/topic/368238-mcs-can-i-schedule-a-machine-catalog-update/

Pro’s

  1. All changes that happened during the VMs uptime, will be lost. From another perspective, this can also be considered very positive. As VMs will stay very clean and will never get different from each other.
  2. In addition to Point 2 from the Con’s: You can work on your master image during work hours, without harming anybody. You can then rollout the image update, but decide for yourself when to initiate the reboot through Citrix Studio.

The Update itself

After the Delivery Group was used sometime with success, we have to bring all those pesky updates that queued up, into the master image.

  1. Boot up your master VM, if it’s offline.
  2. Log in to your master machine via console or RDP.
  3. Apply changes as desired. For example: Windows Updates, Office Updates, Application Updates, Runtime Updates, Citrix VDA Updates and so on.
  4. Run BIS-F to seal and shutdown the master image.
  5. Login to your hypervisor and manually create a snapshot. Give it a meaningful description or some kind of versioning.
  6. Select “Update Machines” inside Citrix Studio:

    Update Machines
    Update Machines
  7. Select the Machine Catalog you want to update:

    Affected Delivery Groups
    Affected Delivery Groups
  8. Select the Snapshot we manually created and prepared beforehand:

    Select updated Master Image Snapshot
    Select updated Master Image Snapshot
  9. Select a Rollout Strategy. As I mentioned before, this is currently a huge bummer for me, as I have no environment, that makes use of Microsoft System Center Configuration Manager (SCCM).
    This means, the only Rollout Strategy I have is: Immediately.
    In my understanding this would translate into: Work each and every Sunday. And I think we agree, that this I a no go.
    For now I have to leave this one unsolved, because the purpose of this blog post is to test, try, learn and study. The obvious answer would be to utilize Citrix PowerShell cmdlets, scripting skills and a Task Scheduler. But this won’t be covered in this blog post.

    Rollout Strategy
    Rollout Strategy

    Important: As I mentioned earlier in this blog post: This is wrong!
    The description from Citrix in this Wizard is not good. If you select “On next shutdown” the whole rollout process will start and work out. The trick is, after the rollout on the hypervisor finished, the reboot has to be initiated through Citrix Studio. More on this topic in a separate upcoming blog post.

  10. Finish the wizard:

    Summary
    Summary

My personal Key findings

  1. Use Citrix Optimizer, not VMware Optimizer, or take your time to study VMware Optimizer real good.
  2. Use BIS-F all the time, it won’t hurt and there is no reason not to.
  3. You may want to create an OU structure in your Active Directory, that let’s you assign different GPOs to your Golden Master Images, then to your active VDA Worker. For example you might want to handle Microsoft Office and Windows Updates different on those machines.
  4. Don’t let Citrix Studio create the VM snapshots for you. Make snapshots yourself and give them meaningful names and a description.
  5. Keep an eye on your snapshot chain. Don’t let it grow too long. Some time in the future you will have to delete old versions.
  6. Make yourself familiar with the concept of MCS rolling catalog updates. Keep two catalog up and running and roll put catalog updates rotational. This is the fastest way back, if you discover business critical errors in your updated machines.
  7. Maybe find a workaround for MCS scheduled machine roll out (no SCCM available). <- Proved wrong. Scheduled rollout is possible via Citrix Studio and PowerShell.
  8. I really need to take a close look at Microsoft Deployment Toolkit (MDT) -> Check out @xenappblog Automation Framework for example.

Upcoming content

Thank you for reading one of the biggest blog posts I ever created. This guides covers the basics to get started with Citrix Machine Creation Services. But there are future topics to get covered.
Blog posts that might follow could be:

  • Citrix Machine Creation Services: Rolling catalog updates
  • Citrix Machine Creation Services: Rollback Machine Catalog
  • Citrix Machine Creation Services: Schedule Machine Catalog updates
  • Citrix Machine Creation Services: Automate the master image update Process
  • Citrix Machine Creation Services: Automate the catalog update

Acknowledgments

A very special Thank You to Jarian Gibson, who helped me a lot with this blog post through the Citrix CTP mentoring program for Citrix CTA. Without him, the information in this blog post would be of less quality.

I would also like to thank René Bigler for proofreading this blog post as he uses MCS a lot longer than me!

This blog article was previously published on mycugc.org:
mycugc.org

Author: Marco

Marco is an IT-System administrator and IT-Consultant with 10+ years experience. He is specialized in the delivery of virtual Apps and Desktops with Citrix solutions. In 2017 he has been awarded Citrix Technology Advocate by Citrix for his community work (#CTA). His second core area is availability & performance monitoring with Zabbix, a leading open-source solution. His employer is the German IT-Company ANAXCO, which is developing a Transport Management Software (TMS) based on Microsoft Dynamics AX. More about Marco

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.