How to evaluate the response time of a Citrix application

This came to my attention on the last few weeks since several projects came with this request and nobody had done this before.
First of all, I’d like to explain a little bit about how the Citrix protocol works on LoadRunner.
Once LoadRunner establishes a connection with the Citrix server, we have two kinds of transactions, Outbound and Inbound. The first comprehending keystrokes and mouse gestures going from the client to the Citrix server. The second comprehend window commands and screen refreshes going from the Citrix server to the client.
Differently from other protocols, were we send a package (HTTP post, WebService call, etc) and wait for the response to measure the response time, the Citrix protocol does not have this kind synchronization. Basically what happens is, we send a keystroke for example, the server responds with a acknowledgment, and that’s it, there is no way of knowing if the applications has processed the request and presented the results. Basically, there is no synchronization between Inbound and Outbound transactions.

In order to measure the response times for Citrix transactions, we need to find a way to “understand” what is coming from the Citrix server and manually synchronize it. There are two ways of doing that, the first is to monitor for window changes, like a window opening, etc. The second is to monitor a part of the screen for a refresh. The problem with both of them is to predict what is going to change, like the name of window that is going open or which part of the screen will refresh. This makes the scripting part really difficult and time consuming, because we cannot predict exactly what is going to be changed, like for example, we’re monitoring for a screen change, but an error message appears, the script will understand that there was a change on the screen and continue, consequently crashing.

Another point to notice is that Citrix is a proprietary solution, based on a widely know protocol (RDP). RDP’s performance is based solely on the bandwidth capacities and being a proprietary solution, we cannot change the protocol, consequently, we cannot increase the performance of the protocol, we can only work with the bandwidth between the Citrix server and the client. This bandwidth varies too much from time to time (peak hours, etc), so the results will vary a lot depending on the time we run the tests.
Based on this assumption, in order to have reliable results, we need to run several rounds of tests during different times, so we can have an idea about the overall performance.

Assuming these problems, I tried to find different ways of getting the response times from Citrix applications. Based on a few articles on the subject, I found that there are more reliable ways of doing that, other than using LoadRunner scripts to capture response times.
Going back to the requirements, we can separate the measure of “response time” to do certain action into three steps. The first is the time it takes from sending a command from the client to the Citrix server. The second is the time the application takes to process that request. The third is the time it takes for Citrix to send the screen refresh with the results to the client. So:

Send Keystroke + Time to Process + Send Screen Refresh = Overall Response Time to Perform an Action

On most cases it is easy and reliable to measure the “time to process” each action locally, without Citrix, so what we need to measure is the time it takes for sending a keystroke to Citrix and the time it takes for the client to receive the screen refresh. Basically is the delay of sending a keystroke and the delay of receiving the screen.

We know that this delays are affected mainly by three factors, the bandwidth limitations, the latency and the amount of data transferred. The first step is to find out how much data is transferred. We can do that by two ways, one is to use a sniffer on the network to capture exactly how much data is sent and received on each transaction. The second is to estimate this amount based on known factors.
For the outbound transactions, this is quite easy, since mouse gestures and keystrokes usually consume the same bandwidth. For the inbound transactions we know that the RDP protocol uses JPG compression to send the screen refreshes, and it only sends the updates for the parts that were changed. This way we can estimate the amount of data transferred by using a simple formula to calculate the JPG size using the image size, density and color content of the refreshed areas.

Knowing the amount of data to be transferred on inbound and outbound transactions we have two options to know the delay. The first is to measure the Round Trip Time (RTT) from the Citrix to the client and the client to the Citrix server, using the amount of data estimated before. The second is to estimate the RTT with known formulas, that use this three factors to estimate the probable delay on the RTT.

After all this technical explanations, I believe that the best solution is to:

  1. Measure the time to process each action locally, without Citrix.
  2. Estimate the average amount of data transferred from the client to the Citrix server, using the standard values for keystrokes and mouse gestures.
  3. Estimate the average amount of data transferred from the Citrix server to the client on the screen refreshes, using captures JPG images of the updates screens.
  4. Measure the RTT with the estimated data for the outbound transactions on different times. This can be easily done by a network engineer once we know the amount of data.
  5. Measure the RTT with the estimated data for the inbound transactions on different times.
  6. Sum all this measures and define our average response time to execute an action over Citrix.

This way we have more reliable results, already on a fine grain, making it easier to identify problems, since we will know if the response times are high on the processing part or the Citrix part.

Note that we’re not trying to measure the Citrix server performance from a load perspective. There are known metrics for Citrix servers that usually satisfy the sizing requirements.

Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+

16 thoughts on “How to evaluate the response time of a Citrix application

  1. I found this post usefully, please can you help with Citrix installation steps

  2. Hello

    How can we measure the time taken to load the first application per server on a citrix environment with broadband or external network in Linux OS
    Do you know any tool which can help in measuring the time/response?

  3. Hi Martin,
    I guess it is pretty late to be posting this since your post is from 2009, but I had a simple doubt regarding performance testing Citrix applications.

    Suppose we need to test a scenario where we have 10 scripts(10 different flows) for X concurrent users. In the case of a normal HTTP server, we would go ahead and test all the flows with X users. This would help us to identify which of these flows show slow response times etc.

    But in the case of citrix, as you have mentioned, the total response time is = Time spent on the AUT server + Time sent for citrix communication.

    So, couldn’t we divide our test in to two parts:
    1) Loadtest the AUT through some other means without Citrix for all flows and identify response times for X users.
    2) Loadtest any one flow for X concurrent users.

    If the application is slow, test (1) will help. If the citrix setup/config has to be tuned, test (2) will help.

    Please tell if I am missing anything here.
    Thanks

    • Hi Krupa,
      From what I understood, that’s almost it. The first step should be right, you could find other ways of testing the target application locally, without using the Citrix protocol. I’m not sure if I understood the second step correctly, but the idea presented on the post suggests no actual load test against the Citrix instance and instead, it suggests a set of simple network measurements to estimate the delay introduced by Citrix. It’s not the standard way of doing a test, but since Citrix scripts are a pain to develop, it might save you some time and still provide meaningful results.

      Cheers,
      Martin

      • Thanks a lot for reply !
        I had mentioned the second step because, it would be good idea to validate our Citrix configuration by running a simple load test. This could give us confidence that the Citrix servers can handle the required load. I would prefer taking this option (running a single script for X users to check citrix performance) as compared to calculating the delay due to citrix. Your thoughts ?

        [Also, if, it is decided to scale the Citrix infra or change some configuration etc. repeating this test(Test 2) could be more quantitative than estimations and calculations]

        Cheers !

  4. Martin,

    This is a great article written in the most simplest form. Although I m looking at at almost a year after it was posted as I was googling for something on ICA RTT. Edgesigh is a great tool to look what’s happenning under the covers. I manage a medium size citrix farm in NA, & encounter people complain about performnace now and then. Without Edgesight it’s would have been so difficult to point out where exactly the problem lies. There is so much data Edgesight captures and there are so many good reports, i m still trying to interpret what the reports mean to the users.

    Thanks
    Vijay.

    • Hi Vijay,
      Glad you found the article useful. Seems like your application is already in production and the users are experiencing some slowness from time to time. Edgesight is a great tool, but it takes some time to set-up, execute and analyze a meaningful scenario. Since your application is already being used and users are noticing the slowness, my suggestion would be to first add some counters (Perfmon) to check the server health (CPU, Memory, IO, Network, etc) while the app is slow. This way you can know if the problem is internal (peak usage) or external (internet connection problem).
      It really takes some time to analyze and find the source of the issue, but again, that’s why we’re here.

      Martin

  5. Hi, It is a good post. All i wanted to know is, with respect to Loadrunner synchronization is the only way to calculate the response time. Is there any way where load runner helps us in finding the response time. for example when an key stroke like “ENTER”, “TAB” it goes faster and it does wait for any event to appear. Likewise when you click on the picklist in an screen it takes time to retrive data from the database and this is recorded as think time in CITRIX protocol. Please don’t mistake these are few of my clarifications.

    • Hi Ram,
      glad you found the article useful and thanks for the comment!

      Apart from this solution, I’ve already used several proprietary tools to test Citrix applications and find response times for specific transactions, including LoadRunner. I like to use this estimation approach to evaluate response times on Citrix when there is no need for hard data and also not much time to record scripts, but sometimes it is necessary to create a simulation, using LoadRunner or any other tool. As I mentioned, the biggest challenge on recording this kind of script is synchronizing the actions. As you mentioned, LoadRunner records the processing time as think time, which is not useful for measuring the real response from the application. In order to measure the correct time, we have to change the recorded think time commands for synchronization commands, like a wait for window or an update on the screen.
      I’ve posted a link to a very interesting article about this issue here. Please have a look and see if it can help you. And keep in mind that it takes some time to get every sync right.. ;-)

      Cheers,
      Martin

  6. Hi, gr8 post thanks for posting. Information is useful!

  7. Hi, very nice post. I have been wonder’n bout this issue,so thanks for posting

  8. I really liked this post. Can I copy it to my site? Thank you in advance.

  9. How to send keystrokes to a CItrix ICA client connection ?
    I am trying through VB but not able to send a key stroke to a ICA session
    Ur idea wil be highly appreciated

    • Hello VenkataKrishnan,
      What worked best for me was EdgeSight. EdgeSight is a tool developed by Citrix just to evaluate Citrix servers performance by scripting actions, reproducing them and collecting metrics.
      I know that there are other tools that can accomplish that, like LoadRunner that you may be familiar with and some C# libraries that can facilitate the connection.
      On LoadRunner the process is pretty straightforward, choosing the Citrix protocol and opening the ICA file to record the actions.
      Last time I’ve done this was using C and not VB, so that may be a problem.

      Thanks for your reply,
      Martin

Leave a Reply