Intro
Performance counters are the types of counters that show you things like CPU and Memory usage as well as granular data like ASP.NET Application Restarts and % Logical Disk Read Time. Various types of performance counters can be viewed by using perfmon.exe. You can add your own custom counters to your project to log various metrics. For example, maybe you want to know how fast a part of your application is running by measuring actions per second.
When you add a custom performance counter, you need to set up a category and a type. The category can be: MultiInstance (1), SingleInstance (0) or Unknown (-1) as seen here. The type number can be one of many different measuring types, for example an amount per second or a total number as referenced here. Once added properly, you can gain new insight into your application and interface with that data in numerous ways through Azure or tools that look into your diagnostics storage.
Setting up the counter in Azure can be somewhat challenging. Essentially, you need to use a startup command with elevated privileges that can add the performance counter to the Cloud Service that you want to monitor. I was not able to get that to work using the OnStart in Azure (only locally); I was able to find a project called: Windows Azure Custom Performance Counters that helped me add the counters in Azure.
Windows Azure Custom Performance Counters is a Visual Studio project that you need to download and build yourself. The result is an executable that needs a reference to Json.NET, you invoke it by passing your custom JSON to through a .cmd file. The JSON represents the category and counter type that you want to add to the Cloud Service.
I advise adding the CreatePerformanceCounters project to your solution if you are or ever intend on using Json.NET. This is because if you just build the .exe and put it in your bin directory with Newtonsoft.Json, you run the risk of using the wrong version and the .exe will fail. You will need to add a custom post build event to copy the compiled .exe to the location for your project’s package. Another option is to use an assembly binding redirect by using a .config file for the .exe that will tell it to use the assembly in the bin directory and prevent constantly copying the compiled .exe. This took some trial and error as well as discussion but eventually the .exe gets compiled correctly each time. The end result from packaging is the .cmd and .exe files in the bin directory of the remote site in Azure. Make sure to put the .cmd (or a copy of it) in a project that is compiled with package and set it to "Copy always" in its properties.
If you run into issues during your deployment, RDP onto the Cloud Service and check that the .cmd file and all its dependencies are in the bin directory of the project you deployed. Modify the counter project to print out the exceptions and include the .pdb file. You can run the .cmd there with Administrator permissions for cmd.exe to see if it worked and work from there if there are issues.
Adding the Counter to the Project
After you have confirmed that you can add the project and deploy, seeing the performance counters on the Azure Cloud Service, you can begin using it in your application. Of course you could test it locally if you want first before deploying, whatever works for you. There are different ways to increment the counters, depending on which one you want to use.
If you want to measure some amount of actions per second for example, just initialize an instance of the PerformanceCounter class with your category name, counter name, no instance name and set it to not be read only. From there you can simply increment that object using a thread-safe call to Increment(). For some metrics you may need to increment by a certain amount of time. You can again read more about the types of counters here.
One thing that I noticed was that some unit tests that hit on the path of the counter would throw exceptions on machines that didn’t have the counter. It’s important to make sure the counter is initialized and accessible in perfmon.exe on the machine that’s running the unit tests if there are tests that call that code. If not, you will see the message: “The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.” This indicates that the counter is not actually installed on the running machine; you can install it the same way on the Cloud Services as on the machine running the test code. You may need to start PowerShell again (if you add the counters and then execute the tests in the same script) because the PowerShell session can cache the counters if it adds them first.
A better approach is to make an interface for the counter and to fake that interface. That way you can make a call to the increment always return an expected value. This way you do not need to mess with your local system and add the performance counters.
Monitoring the Counter
After you have finally added the monitor and invoked it in the code, you can begin monitoring it. You can go directly onto the Cloud Service and look at the counter using perfmon.exe; you can use a third-party tool like Cerebrata’s Azure Management Studio or you can even use the Azure Portal.
If you want to use the Azure Portal to monitor the metrics, you need to configure the whole Cloud Service to log with Verbose and save that. From there you can go into the monitoring section and find your counter(s). Simply add the counters to the monitor and you can watch the metrics right in the portal.
Summary
Setting up counters and configuring the machines running tests and in Azure running live can be bit of back and forth due to the paths, permissions and category types. Once you have it set up though, it can give you useful information. You may way to explore some other articles on knowing the right categories, the optimal times to transfer the metrics and other concerns that may arise.
Useful Links
- An Introduction To Performance Counters
- Performance Counters-MultiInstance/MultiCategory
- Using performance counters in Azure
- Windows Azure Custom Performance Counters
- Performance Counter Types
- Performance Counter Categories
