Taking Memory Leak Traces

This post talks about the steps to take traces for a BizTalk host as well as memory leak traces for host process. Memory leak traces applies to any windows process. These traces are normally required when you are having performance issues especially due to memory leaks. Pssdiag trace for BizTalk is provided by Microsoft support team usually to troubleshoot issues. Normally, you will work with Microsoft support to take traces and provide them to analyze the traces. These steps are just a guideline and help you prepare better in some cases beforehand.

Before start

  1. Make sure host that has memory leak issue is not currently consuming high memory (restart and wait for 5 min if needed)
  2. Start BTS pssdiag (with BTStrace, MBV, and perfmon) (See below)
  3. Trigger 1 manual userdump during normal memory usage (See below))
  4. Set up Debugdiag leaktracker with memory dumps (see below)
  5. Wait for memory to get high
  6. Trigger 2 manual memory dumps (even if leaktracker already took some)
  7. Download and install DebugDiag from here – https://www.microsoft.com/en-us/download/details.aspx?id=49924

Steps for PSSDiag:

  1. Extract the PSSDiag zip on all the BizTalk servers in the group. Follow below steps on all machines.
  2. Open Initialize.exe as an administrator and have the following settings in place. Change the “Host Instance To Track” to host for which you see the issue. Don’t select the WCF trace option.
  3. Open up a command prompt as an administration and go to the PSSDiag folder.
  4. Run “Pssdiag /c” to start the trace. Wait for the collection started message to appear.
  5. Wait for the issue to happen. Stop the trace once you start seeing the issue in your environment. Make sure you capture at least one occurrence of the issue.
  6. Stop the PSSDIag trace by pressing Ctrl + C. Wait for the stop to complete.
  7. Provide the Output folder present in the PSSDIag folder.

Debugdiag leaktracker with memory dumps:

  1. Open DebugDiag, Add Rule, and choose Native (non-.NET) Memory and Handle Leak and click Next
  2. Select the BizTalk process that has the memory issue and click Next
  3. Under Userdump generation, click the Configure button
  4. In the Configure userdumps window, make sure Auto-create is checked
  5. Check Generate userdump when private bytes reach 700mb and each additional 200mb
  6. Check Generate userdump when virtual bytes reach 900mb and each additional 300mb
  7. Click Save & Close and click Yes to the “Are You Sure” message
  8. Click Next and select a Userdump Location with plenty of space (preferably over 20gb of free space) and click Next
  9. Choose Activate the rule now and click Finish
  10. If you get a popup asking about setting Symbol Search Path, click Yes

The above action plan assumes the normal memory usage is below 700mb for this host and it goes above 900mb before the perf issue happens (since we want atleast 2 dumps).

If the above statement is not true, we may need to tweak these settings.

If we’re not sure about what memory usage looks like before and during the issue, we can go with the above settings on the understanding that we may need to tweak them in the future if the necessary dumps are not captured on failure.

Collect Manual Userdump

  1. Open DebugDiag collection.
  2. Hit cancel on the first screen.
  3. Click on the “Processes” tab on the next screen.
  4. Select the BizTalk process from the list of processes.
  5. Right click the process and select “Create Manual Dump” option.

One thought on “Taking Memory Leak Traces

  1. Pingback: March 30, 2020 Weekly Update on Microsoft Integration Platform & Azure iPaaS | Hooking Stuffs Together

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s