Have you ever wondered why Forefront TMG Client processes (FwcAgent.exe, FwcMgmt.exe) show up on Task Manager as 32bit processes as shown in Figure below?

Many Firewall Admins start wondering why this happens even when they know that Forefront TMG Client is a full 64bit application. The answer is: this is an expected behavior. The Forefront TMG Client package supports both 32- and 64-bit versions of Windows and on 64-bit Windows supports both 32- and 64-bit processes. The FwcMgmt.exe and FwcAgent.exe processes do not use any of the capabilities that are only available for 64-bit processes (e.g. larger memory space) so it was decided that having both 32-bit and 64-bit flavors is not worth the effort. However, it is important to notice that Forefront TMG Client includes a WinSock Layered Service Provider component, which is loaded inproc by WinSock applications and therefore must have the same “bit-ness” as the application. Hence there are two WinSock Layered Service Provider DLLs, FwcWsp.dll and FwcWsp64.dll, supporting 32-bit and 64-bit applications, respectively:

The “bit-ness” of the FWC processes (management tools) does not have to match the “bit-ness” of any application and hence having them 32-bit is good enough.

Author
Oved Itzhak
Senior SDE
Microsoft Forefront TMG Team

Technical Reviewers
Yuri Diogenes
Sr Security Support Escalation Engineer
Microsoft CSS Forefront Security Edge Team

Ori Yosefi
Senior Program Manager
Microsoft Forefront TMG Team

We would like to inform you that we have released two new rollups.

Forefront Threat Management Gateway (TMG) 2010 Service Pack 1

We have released Software Update 1 Rollup 2. More information about the rollup is available in the following KB article- http://support.microsoft.com/kb/2475183

ISA Server 2006 SP1 Hotfix Package

We have released a Hotfix a Hotfix Package for ISA Server 2006 SP1. More information is available in the following KB article - http://support.microsoft.com/kb/2475184

Availability

If you are affected by one of the symptoms described in the articles above, you can request access to hotfix by pressing a link in the article (similar to the one in the picture below):

These fixes will also be included in the next service pack released for these products.

Seasons greetings and happy new year,

The Forefront Threat Management Gateway team

Introduction

Consider a scenario where users behind ISA Server (internal network) start to receive random prompts for authentication while trying to access internet using ISA Server as proxy. The authentication prompt persists even after entering the credentials. To resolve the issue it is necessary to restart Firewall Service.

Although you probably heard or read about this scenario many times, the goal of this post is to give you a compiled version of the action plan and what to look for while analyzing the data.

Data Collection

Start by following the plan from this post (basics section), along with that make sure that binding order is also correct i.e. internal NIC is higher in order then the external. Wrong binding order can cause issues such as the one mentioned here. In addition to the data gathering specified previously, also collect the following data:

1. Use ISA Data Packager while doing repro of the issue.
2. Enable netLogon logging on the ISA server nodes, using command nltest /dbflag:0x2080ffff in the command prompt as per KB109626.
2. Set the Performance counters as specified in this post.

Data analysis

When start reviewing the perfmon data you want to check the counter ISA Server Firewall Packet Engine\Backlogged Packets. You will notice a trend similar to the perfmon screenshot showed in this post. This can happen due name resolution issue as explained in this TechNet Article.

Next data to analyze is the netlogon.log, which also can be done using the same approach as the following post. In other words, look for the following pattern:

08/21 12:00:00 [DOMAIN] Contoso: Domain thread started 08/21 12:00:00 [DOMAIN] Contoso: Domain thread started doing API timeout 08/21 12:00:00 [SESSION] Contoso: Contoso: NlTimeoutApiClientSession: Unbind from server \\ab-cd.Contoso.local (TCP) 0.

From above data it appeared we can conclude that the Domain Controller to which ISA server had the secure channel established with, did not responded in time manner, which triggered the NlTimeoutApiClientSession in the netlogon logging. After that ISA Server resets the secure channel and tries to make secure channel with another DC.

Resolution for this Particular Case

In this particular case the clients were using WPAD (automatic detection), which by default returns the IP address of the ISA Server rather than the name. This forced the client to use NTLM authentication rather than Kerberos (supported in IE7 or higher).

In order to force WPAD to use FQDN instead of IP address we ran the script described in this post. After running the script, all the web proxy clients using WPAD started getting FQDN of the ISA server nodes and use Kerberos for authentication, which enhance the authentication traffic and decrease the number of authentication request.

Author
Suraj Singh
Security Support Engineer
Microsoft CSS Forefront Security Edge Team

Technical Reviewer
Yuri Diogenes
Sr Security Support Escalation Engineer
Microsoft CSS Forefront Security Edge Team

1. Introduction

When dealing with ISA high CPU utilization where wspsrve.exe is the one consuming more resources, the first impression is that ISA is the culprit for that. There are some scenarios where this statement is true, such as this one that I documented last year. But there are other scenarios where this is not true, including scenarios where wspsrv.exe is crashing – not always is because of ISA itself. I tried to demystify this on this post that I wrote back in 2009.

This post is about a scenario where ISA Server was facing a 100% CPU utilization and wspsrv.exe process was consuming most part of that (around 93%). This was happening at least twice a day and the only way to get CPU get to a normal utilization was by restarting Firewall Service. Sounds like an ISA issue right? Not really and you will see.

2. Gathering Data

Use the same data gathering approach as the one listed in this post with an addition of getting a full memory dump using notmyfault tool, see KB972110 for more info.

3. Analyzing the Data

By reviewing perfmon data from the time that the issue was happen it was possible to see that wspsrv.exe was 88% average of CPU utilization, but notice that within this process there is no thread spending too much CPU cycles as shown below:

In order to better understand this issue we started to review the manual dump of wspsrv.exe process. By reviewing the user mode dump we were unable to determine what the root cause of the issue was. Most of the threads were waiting for something in Kernel mode as shown below:

> First step was to review the top threads that had calls in Kernel mode:

0:000> !runaway 7

 User Mode Time

... [hiding the values as they are not relevant here]

...

...

 Kernel Mode Time

  Thread       Time

  69:1310      0 days 0:00:49.500

  53:129c      0 days 0:00:46.812

 104:113c      0 days 0:00:42.390

  45:127c      0 days 0:00:40.875

  78:688       0 days 0:00:40.734

  94:4d0       0 days 0:00:38.890

 162:18a8      0 days 0:00:38.500

 166:18c8      0 days 0:00:38.093

 108:1418      0 days 0:00:37.062

> By taking a closer look into the threads we see that some of them are either performing RPC or DNS such as the one below:

0:000> ~104kb

ChildEBP RetAddr  Args to Child             

WARNING: Stack unwind information not available. Following frames may be wrong.

28cada54 77c7feb0 28cada90 28cada74 77c80845 ntdll!KiFastSystemCallRet

28cada60 77c80845 28cada90 76ed42d8 28cade84 rpcrt4!I_RpcSendReceive+0x23

28cada74 77ce415a 28cadabc 24b86ff4 00000000 rpcrt4!NdrSendReceive+0x28

28cade64 76ed5049 76ed42d8 76ed421c 28cade84 rpcrt4!NdrClientCall2+0x1a8

28cade7c 76ed4f69 00000000 29e2f078 00000001 dnsapi!NetInfo_Copy+0x5c2

28caded8 76ed6f5d 29e2f078 00000001 00000000 dnsapi!NetInfo_Copy+0x4e2

28cadf0c 76ee9d0c 00000003 28cae1d8 00000001 dnsapi!DnsValidateName_W+0x31f

28cadf34 62ea2cc4 28cae1d8 00000001 00000000 dnsapi!DnsQuery_A+0x20

28cadf78 62ea30a2 28cae1d8 00000001 28cadfb8 msphlpr!ProxyDnsCacheInit+0x52f

28cadf98 62ea217c 28cae1d8 28cadfb8 62ea1f0f msphlpr!ProxyDnsCacheInit+0x90d

28cae2dc 64760c2a 021d7710 28caeedc 00000001 msphlpr!ProxyGetHostByName+0x26d

28cae764 64761d11 28caeed4 00000400 28caee74 W3Filter!DllUnregisterServer+0x42ca6

28cae9d4 647638dc 28caeed4 00000400 28caee74 W3Filter!DllUnregisterServer+0x43d8d

28caf534 647295c7 00000000 00000000 28cafe14 W3Filter!DllUnregisterServer+0x45958

28caf554 647262ef 021d7008 00000000 000003e5 W3Filter!DllUnregisterServer+0xb643

28caf574 647277f8 28cafe14 00000000 00000001 W3Filter!DllUnregisterServer+0x836b

28caf590 647281c5 28cafe14 28038ce0 28038728 W3Filter!DllUnregisterServer+0x9874

28cafe1c 64728ad0 000000a7 00000000 0000000c W3Filter!DllUnregisterServer+0xa241

28cafe90 0046051a 28038728 00000001 00000000 W3Filter!DllUnregisterServer+0xab4c

28cafefc 0046032d 000000a7 00000000 00000001 wspsrv!SSL_CREDENTIALS::operator=+0x2a870

28caff20 00460bea 2685d018 004602b1 28caff50 wspsrv!SSL_CREDENTIALS::operator=+0x2a683

28caff30 00460266 000000a7 00000000 00000001 wspsrv!SSL_CREDENTIALS::operator=+0x2af40

28caff50 0044b0ff 2685d0bc 000000a7 00000000 wspsrv!SSL_CREDENTIALS::operator=+0x2a5bc

28caff7c 0044c08e 2685d0bc 000000a7 00000000 wspsrv!SSL_CREDENTIALS::operator=+0x15455

28caffb8 77e6482f 0000004f 00000000 00000000 wspsrv!SSL_CREDENTIALS::operator=+0x163e4

28caffec 00000000 0044bf26 0000004f 00000000 kernel32!GetModuleHandleA+0xdf

It was time to review the kernel dump file and verify if we had more information there:

> In this case we first started reviewing if there was anything locked in kernel space:

1: kd> !locks

**** DUMP OF ALL RESOURCE OBJECTS ****

KD: Scanning for held locks............................................................

Resource @ 0x8a7f3008    Shared 1 owning threads

    Contention Count = 856651

    NumberOfExclusiveWaiters = 5

     Threads: 84149020-01<*>

     Threads Waiting On Exclusive Access:

              84166a80       8961d620       84166750       89649240      

              841777a0      

> Notice that we have 5 exclusive waiters wanting on a thread. Let’s see which thread is that:

1: kd> !thread 84149020

THREAD 84149020  Cid 1074.18a8  Teb: 7ff12000 Win32Thread: 00000000 READY

IRP List:

    fe5eae28: (0006,01d8) Flags: 00000884  Mdl: 00000000

    88bb1a68: (0006,01d8) Flags: 00000000  Mdl: 89128478

    fee50ba8: (0006,01d8) Flags: 00000000  Mdl: 880eb6c8

    f9d62af8: (0006,01d8) Flags: 00000000  Mdl: 887adee8

    fa284008: (0006,01d8) Flags: 00000000  Mdl: 87fb7bb8

    feb2dd80: (0006,01d8) Flags: 00000000  Mdl: 88bbdc48

    ff8c95f0: (0006,01d8) Flags: 00000000  Mdl: 88bbe488

    fac4b820: (0006,01d8) Flags: 00000000  Mdl: 8847d798

    fea1d008: (0006,01d8) Flags: 00000000  Mdl: 89252968

    ff097008: (0006,01d8) Flags: 00000000  Mdl: 8a5ee320

    fe64b008: (0006,01d8) Flags: 00000000  Mdl: 88a114a0

    feba1b30: (0006,01d8) Flags: 00000000  Mdl: 8975beb8

    ff499550: (0006,01d8) Flags: 00000000  Mdl: 87cd53a8

    ff5cf9f0: (0006,01d8) Flags: 00000000  Mdl: 8856dd18

    fc26ec38: (0006,01d8) Flags: 00000000  Mdl: 883df0a8

Not impersonating

DeviceMap                 e1750820

Owning Process            89f4ead8       Image:         wspsrv.exe

Wait Start TickCount      4518588        Ticks: 0

Context Switch Count      3718083            

UserTime                  00:01:15.546

KernelTime                00:01:02.906

Win32 Start Address 0x0044bf26

Start Address 0x77e617ec

Stack Init b7b39000 Current b7b38898 Base b7b39000 Limit b7b36000 Call 0

Priority 8 BasePriority 8 PriorityDecrement 0

ChildEBP RetAddr  Args to Child             

b7b388b0 8083d5b1 84149020 841490c8 00000000 nt!KiSwapContext+0x26 (FPO: [Uses EBP] [0,0,4])

b7b388dc 8083df9e 84149020 8a7f3008 00000000 nt!KiSwapThread+0x2e5 (FPO: [Non-Fpo])

b7b38924 8082e057 88f99718 0000001b 00000000 nt!KeWaitForSingleObject+0x346 (FPO: [Non-Fpo])

b7b3895c 80824ba8 89a0a400 00000000 8082c38e nt!ExpWaitForResource+0x30 (FPO: [Non-Fpo])

b7b3897c 8085110e 8a7f3008 00000001 b7b389d4 nt!ExAcquireResourceSharedLite+0xf5 (FPO: [Non-Fpo])

b7b3898c b9fb5c70 8a7f3008 89a0a400 00000000 nt!ExEnterCriticalRegionAndAcquireResourceShared+0x19 (FPO: [Non-Fpo])

b7b389d4 b9fb5b17 b7b38a18 b7b389ec 00000028 afd!AfdGetTransportInfo+0x1b (FPO: [Non-Fpo])

b7b389f0 b9fb5d9f b7b38a34 b7b38a18 00000000 afd!AfdAllocateEndpoint+0x1e (FPO: [Non-Fpo])

b7b38a38 b9fbf7a4 89b1e438 8a9a9f38 b7b38a5c afd!AfdCreate+0x13b (FPO: [Non-Fpo])

b7b38a48 80840153 8a9bc030 fe5eae28 fe5eae28 afd!AfdDispatch+0x79 (FPO: [Non-Fpo])

b7b38a5c 8092e87e b7b38c04 8a9bc018 00000000 nt!IofCallDriver+0x45 (FPO: [Non-Fpo])

b7b38b44 8092c3ea 8a9bc030 00000000 89062228 nt!IopParseDevice+0xa35 (FPO: [Non-Fpo])

b7b38bc4 8092d80d 00000000 b7b38c04 00000042 nt!ObpLookupObjectName+0x5b0 (FPO: [Non-Fpo])

b7b38c18 8092c6cd 00000000 00000000 dffa7c01 nt!ObOpenObjectByName+0xea (FPO: [Non-Fpo])

b7b38c94 80931d92 29b2f958 c0140000 29b2f90c nt!IopCreateFile+0x447 (FPO: [Non-Fpo])

b7b38cf0 8092fb28 29b2f958 c0140000 29b2f90c nt!IoCreateFile+0xa3 (FPO: [Non-Fpo])

b7b38d30 80833bef 29b2f958 c0140000 29b2f90c nt!NtCreateFile+0x30 (FPO: [Non-Fpo])

b7b38d30 7c82860c 29b2f958 c0140000 29b2f90c nt!KiFastCallEntry+0xfc (FPO: [0,0] TrapFrame @ b7b38d64)

WARNING: Frame IP not in any known module. Following frames may be wrong.

29b2f9ec 00000000 00000000 00000000 00000000 0x7c82860c

> This stack has a very good amount of IRPs (I/O Request Packet), let’s see if there is any pending IRP:

1: kd> !irp 88bb1a68

Irp is active with 4 stacks 2 is current (= 0x88bb1afc)

 Mdl=89128478: No System Buffer: Thread 84149020:  Irp stack trace. 

     cmd  flg cl Device   File     Completion-Context

 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

                     Args: 00000000 00000000 00000000 00000000

>[  f, 8]   0 e1 8a94ddf0 870404b0 b9e964d0-870f66c8 Success Error Cancel pending

              \Driver\Tcpip*** ERROR: Module load completed but symbols could not be loaded for idmtdi.sys

       idmtdi

                     Args: 0000127b 00010020 00000000 00000000

 [  f, 8]   0 e0 8a85d020 870404b0 b9fc1b72-833955a0 Success Error Cancel

              \Driver\IDMTDI      afd!AfdRestartBufferReceiveWithUserIrp

                     Args: 0000127b 00010020 00000000 00000000

 [  e, 5]   5  0 8a9bc030 8583a190 00000000-00000000   

              \Driver\AFD

                     Args: 0000127b 00000000 20000020 00000000

> Notice that there is a third party transport driver interface (idmtdi.sys) loaded within this IRP and it is pending.

4. Conclusion

As you could see not always the user mode dump will give us the definitive answer for a problem, mainly when most of the threads on user space are waiting for something to happen on kernel space. In this particular case, when reading the kernel dump, we used the !locks command, which displays all locks held on resources by threads. A lock can be shared or exclusive, which means no other threads can gain access to that resource. In this case we saw that we had five threads waiting for exclusive access. The threads that were waiting for exclusive access were part of the wspsrv.exe.

What we can conclude here is that we have a condition where wspsrv.exe was allocating more and more threads without releasing resources (releasing threads) because it was getting stuck waiting for an answer from kernel driver. Some threads were working on I/O, mainly socket and LRPC call (almost socket) that run in Kernel mode. The third party driver is causing every AFD call to take more time and more CPU cycles, the fact that ISA is extremely using sockets amplify the problem. The third party kernel driver was uninstalled from ISA Server and the issue was resolved.

Another takeaway from this particular scenario is that not always that wsprsv.exe is using lot of CPU resources is because it’s an ISA issue. Keep that in mind.

Author

Yuri Diogenes

Sr. Support Escalation Engineer

Microsoft CSS Forefront Edge Team

Technical Reviewer

Franck Heilmann

Sr. Escalation Engineer

Microsoft CSS Forefront Edge Team

Hi all,

We thought it could be useful do create a series of blog posts about TMG basics (i.e. TMG 101). This would cover the different TMG features and concepts and walk people who are new to TMG through the first steps.

We thought of creating a new blog (called “TMG from the grounds up”) for these posts but we heard from a few people that they would rather that we post these articles in this blog (i.e. the TMG product team blog).

What do you prefer (please leave a comment)? Should we create a new blog, post to this one or both?

Thanks in advance for your feedback,

Ronit and Ori from the TMG team

The Microsoft TMG/UAG Server support team in US is looking to hire a Full Time Employee. If you’d be interested to come and work with us and learn more about TMG/UAG internals than you probably would anywhere else in the world ,then this might be the job for you. We’re looking for someone very experienced who can hit the ground running and could be expected to work the majority of our easier support cases without too much assistance. Therefore you’d need a minimum of 5 years’ experience (preferably more) working with ISA/TMG/UAG Server day to day in an administrative capacity (probably as a production administrator or developer ).

Specialties of particular interest to us would be:

• Setup / configuration / migration
• Administration of ISA/TMG and ideally IAG/UAG
• Good Network skills like reading network traces and such
• Crashes, Hangs and Unexpected behavior

Beyond that if you have experience of other Microsoft server products (Windows, SharePoint, Exchange), windbg, c# or c++, it won’t do you any harm. If you think you’d be interested, please apply to the positions below:

https://careers.microsoft.com/JobDetails.aspx?ss=&pg=0&so=&rw=1&jid=34779&jlang=EN
https://careers.microsoft.com/JobDetails.aspx?ss=&pg=0&so=&rw=1&jid=34780&jlang=EN

Recently Microsoft Customer Service and Support have seen cases with the firewall process (wspsrv.exe) in ISA Server 2006 “leaking” memory until it reaches the 32-bit process limits or face too much memory fragmentation to continue working properly.

On an ISA Server with high loads, memory fragmentation can always cause issues and that’s why we recommend setting the HeapDeCommitFreeBlockThreshold registry key on high load systems. This is to improve how the heap manager decommits the memory and to minimize the heap fragmentation. This is described in detail in http://support.microsoft.com/kb/315407.

However even though setting this key, can help to improve the overall stability and performance of your server, it will only be a temporary workaround if you’re facing a leak inside the process. In a nutshell, a memory leak is memory allocated in a process that is not freed at a later point in time. If enough memory allocations like this happen, the process may run out of memory.

For more details about memory leaks I suggest that you have a look at some of Mark Russinovich’s blog posts like “Pushing the Limits of Windows: Virtual Memory” or recorded sessions around “Pushing the Limits of Windows” which can be found here: http://technet.microsoft.com/en-us/sysinternals/bb963887

A leak inside the ISA firewall process (wspsrv.exe) can be caused by many things. E.g. we’ve seen issues with 3^rd party Add-Ins, drivers, and unexpected behavior in our application. Identifying the cause of a leak can be a very complex troubleshooting task and in most cases you’ll need to collect Performance Monitor logs to see how the allocated memory is behaving. E.g. how much does it increase by per day, does it ever decrease etc. If the memory allocations keep increasing over time, then this may be a sign of a leak and it may be necessary to collect LeakTrack memory dumps using DebugDiag and analyze the dumps for leaks.

In order to avoid you having to do this, we’re doing our best to identify and fix the leaks we find and to include these fixes in regular software updates.

As mentioned at the beginning, I want to let you know about one issue we identified recently. The issue we identified is unfortunately introduced in an optional update of the winhttp.dll for Windows 2003 x86 (http://support.microsoft.com/kb/971737). This update implements EAP Authentication in Windows HTTP services. There’s no need to install this update on ISA Server 2006, because we don’t use this feature of the winhttp.dll.

The leak occurs when you configure HTTP connectivity verifiers in ISA that verify HTTP connections to web servers that use Windows Integrated (NTLM/Kerberos) authentication.

To prevent running into this issue, please either uninstall the optional update from KB971737 or review your connectivity verifiers, and check whether any of them verifies against an authenticating HTTP server. If so, disable or move to an anonymous server. Please note that HTTP connectivity verifiers may also be used in Web Farms.

If you’re not sure if the update is installed on your ISA server, then please open your Control Panel -> Add Remove Programs -> Show Updates and review the list of installed updates

Or open Accessories -> System Tools -> System Information -> Loaded Modules

Verify the version number of winhttp. The version number of the leaking dll is 5.2.3790.4584.

Summary

Please don’t install KB971737 on computers running ISA Server. If it’s installed, please uninstall the fix if you’re using Connectivity verifier on Webserver with Integrated Authentication.

Author

Philipp Sand
Microsoft CSS Forefront Security Edge Team

Technical Reviewer

Lars Bentzen

Microsoft CSS Forefront Security Edge Team

If you read the article Economics of the Cloud published last November on Microsoft on the Issues blog, you will see the that Microsoft analysis “uncovers economies of scale for cloud that are much greater than commonly thought”. As more and more business start to move to the cloud there is also the aspect of secure cloud access by an on premises gateway. A new article written by Yuri Diogenes (a former Senior Support Escalation Engineer from Forefront Edge CSS Team) was recently published at TechNet Magazine (February 2011 issue) and explains how Forefront TMG 2010 can assist you during this deployment.

Read the full article here: http://technet.microsoft.com/en-us/magazine/gg607680.aspx

Hi all,

Our very prolific Yuri Diogenes just contributed a new WIKI page called “Forefront TMG 2010 Survival Guide” (available here) which you may find interesting.

The WIKI pages are “live” documents and we welcome additions, edits and contributions.

Some information about using the WIKI initiative is available here:

http://social.technet.microsoft.com/wiki/contents/articles/wiki-how-to-join.aspx

http://social.technet.microsoft.com/wiki/contents/articles/wiki-getting-started.aspx

http://social.technet.microsoft.com/wiki/contents/articles/wiki-how-to-contribute.aspx

http://social.technet.microsoft.com/wiki/contents/articles/wiki-code-of-conduct.aspx

One note: If you think Yuri had enough, think again. He is working on a “Troubleshooting Forefront TMG survival guide” so:

1. You have something to wait for

2. Should wait before contributing a troubleshooting article

Microsoft Windows 2008 Server R2 SP1 has recently been released. SP1 contains changes that are focused on supporting new types of hardware,
adding support for several emerging standards, and addressing specific reliability, performance, and compatibility issues.
SP1 changed areas relevant to TMG/UAG products are below:

TMG 2010 SP1 and UAG 2010 SP1 are fully compliant with the new Windows service pack.

We have successfully tested both TMG and UAG scenarios after the upgrade to new Windows service pack.
Moreover, installing TMG/UAG on Windows 2008 R2 SP1 machine is also supported.

When upgrading gateway machine to Windows 2008 R2 SP1, restart will be required.

After installation, the TMG/UAG services restart automatically.

We suggest upgrading TMG SP1 in the following order:

1. Enterprise Management Servers (master and replicas – only for TMG deployments).

2. Array managers.

3. Array members.

For deployments with NLB enabled, we encountered one issue:
After upgrading to Windows 2008 R2 SP1, creating new NLB cluster using the administration console may fail.
Machine network interface of the newly load balanced network will become disabled. 

As a workaround, we suggest rebooting the TMG/UAG machines after SP1 installation once again.

This will ensure that setting up new NLB cluster after Windows upgrade will succeed.

Author: Igor Zarivach, Software Design Engineer

Reviewer: Eyal Zangi, Senior Test Lead

The tireless Yuri Diogenes was interviewed on Talk TechNet last week. If you did not get a chance to hear it live, a recording is available here.

Yuri talked about Forefront TMG and answered a lot of questions. Highly recommended.

With TMG we introduced a feature called “Enhanced NAT” (ENAT). ENAT enables you to configure the IP address you want to use as source IP when you NAT the traffic between 2 networks. The IP address has to be configured on the Interface, which is connected to the Destination Network, before you can select it as source IP.

The default configuration of a Network Rule with NAT in TMG is to “Use the default IP address” of the network adapter as source address. With this configuration TMG behaves similar to ISA Server.

I didn’t say that TMG behaves the same way as ISA in this regards, because the difference between ISA Server and TMG in this case can be very important in some scenarios.

ISA Server did use the IP which had been configured in the first properties page of your network adapter TCP/IPv4 Properties:

In this case ISA would always use the 192.168.1.223 as source address when you configured a NAT relationship.

With TMG and ENAT things slightly change…

For all traffic handled by the Web Proxy Filter, TMG will still use the 192.168.1.223 in this scenario. It’s not possible to change this behavior because of the internal design of the Web Proxy Filter.

Please be aware, that the TMG Web Proxy Filter behavior in this regards doesn’t depend and cannot be modified by any Network Rule setting you configure.

For all other protocols, TMG will not always use the IP which had been configured on the “main property” page of the TCP/IPv4 settings, BUT TMG will use the smallest IP configured on this interface if you choose the “Use the default IP Address” setting.

Whats the smallest IP?

Consider the scenario, where you did configure the 192.168.1.223 on the main property page, and afterwards add the IP 192.168.1.222 in the Advanced Settings:

On the first TCP/IPv4 property page in Windows, you’ll still see the 192.168.1.223.

When you open the TMG MMC and open Networking -> Network Adapters, you’ll see this:

As 222 is smaller than 223, TMG chooses the 222 as Default IP.

To be aware of this change of the Default IP Addresses selection is very important, in scenarios with IPSec Site-2-Site where you expect the traffic with specific Source IP addresses, or in scenarios where the routing or firewall setup “expects” specific IP addresses in order to properly handle the traffic.

If you configure the NAT rule using the “Default IP address”, and add another smaller IP to your interface, TMG will automatically use the smaller IP address as source IP. If you didn’t modify your infrastructure configuration, this change can break IPSec connections or accidently block traffic send from TMG on the next Firewall, where you didn’t expect this change in IP.

Best Practices for ENAT

1. Avoid using the Default IP Address Setting when configuring NAT rules for interfaces with multiple IP addresses, or where you plan to add additional IP addresses later.

Try to use the “Use the specified IP address” or “Use multiple IP addresses” NAT Address Selection settings, whenever possible.

2. If you want to use the same “NAT” IP for Web Traffic handled by the Web Proxy Filter and for all other protocols, select the IP address specified on the first TCP/IPv4 property page when configuring a specified IP address.

Why did this behavior change from ISA to TMG?

The main cause for this change in behavior is a change in the Windows Pubic API GetAdaptersAddresses (http://msdn.microsoft.com/en-us/library/aa365915(VS.85).aspx ) which returns the IP addresses from the structure IP_ADAPTER_ADDRESSES (http://msdn.microsoft.com/en-us/library/aa366058(v=vs.85).aspx ) to determine the first IP address of an interface.

From the description of IP_ADAPTER_ADDRESSES we can see, that this structure changed from 2003 to 2008 or 2008 R2:

Quote:

The order of linked IP_ADAPTER_UNICAST_ADDRESS structures pointed to by the FirstUnicastAddress member that are returned by the GetAdaptersAddresses function does not reflect the order that IP addresses were added to an adapter and may vary between versions of Windows

This means that TMG only relies on the order of IP addresses returned by GetAdaptersAddresses, and will consider the first IP found in the list as the Primary IP (from a TMG perspective).

Author

Philipp Sand
Microsoft CSS Forefront Security Edge Team

Technical Reviewer

Eric Detoc

Microsoft CSS Forefront Security Edge Team

The user activity report is a new feature of Forefront TMG SP1. I recently came across an issue where the customer was trying to run TMG user activity reports. When he used a single user (domainname\username) it rendered the report okay, but when we tried the same for 2 users (domainname\username1; domainname\username2) it gave the following error message after 5 minutes:

As a part of the troubleshooting we tried to use the underlining SQL Server Reporting Server (SSRS) to run the same reports. This worked and we did not run into the issue.From looking at the built-in tracing in TMG, we identified that TMG failed to render the report because the operation timed out. By default the timeout for this report is 5 minutes.To fix the issue we created the following registry key:

Please note you may have to create the registry keys as they do not exist by default:

0x000927c0 (HEX) =600000 milliseconds = 10 minutes.
The default timeout value is 5 minutes.

This worked for 2 users, but with 10 users it still timed out and we got the same error. We eventually fixed the issue by increasing the registry value to 20 minutes. The main reason why we had to increase the timeout value was because the machine was not powerful enough to complete the processing of multiple users within the default 5 min.

Author
Saurav Datta
Security Support Escalation Engineer
Microsoft CSS Forefront Security Edge Team

Technical Reviewer
Lars Bentzen
Escalation Engineer
Microsoft CSS Forefront Security Edge Team

As I mentioned in a previous post, Yuri Diogenes was working on a TMG troubleshooting survival WIKI page. The page is now ready and is available here.

Reminder: WIKI pages are  “live” documents and you are welcome to contribute edits to them. Knowing Yuri – you’re going to have a hard time finding a trick he doesn’t know.

Introduction

Consider the scenario where we have URL Filtering enabled on TMG 2010 Server and it is not working.

Troubleshooting

A quick look at the Alerts section in TMG MMC shows:

The failure is due to error: A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider.

Description: An error occurred while trying to communicate with the Microsoft Reputation Service server. If this Forefront TMG server is chained to an upstream server, verify that the WinHTTP proxy is set to localhost. If this issue persists, check that Internet connectivity is available.

In our case we were able to access the Internet properly and all other settings looked good.

As a troubleshooting step we accessed the MRS Websites https://10.ds.mrs.microsoft.com and https://10.ts.mrs.microsoft.com from TMG Server and we received an error that the certificate is not trusted.

We discovered that the root certificate from “GTE CyberTrust Global Root” was missing from the server.

In order for URL Filtering to work properly we need to have trusted root certificate installed on TMG Server.

After installing “Update for Root Certificates Update” the issue was resolved.

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=25249786-2B8E-4C51-8F4B-727CE25CC2C5

Conclusion:

URL Filtering is a cloud based service and must be able to successfully establish an HTTPS connection to the either of the MRS (Microsoft Reputation Service) sites mentioned above.

http://technet.microsoft.com/en-us/library/ee869543.aspx

Authors

Junaid Ahmad Jan

Security Support Engineer,

Microsoft CSS Forefront Security Edge Team

Technical Reviewer

Richard Barker

Sr. Security Support Escalation Engineer,

Microsoft CSS Forefront Security Edge Team

Introduction

Recently we’ve encountered a number of cases where customers wanted to use TMG to require strong authentication for some parts of a published web site (e.g. Outlook Web Access, OWA) but not for others (e.g. Exchange Active Sync, EAS).

This post will describe how to configure TMG for similar scenarios.

Background

The TMG authentication process, as described here, has three phases:

• Receipt of client credentials.
• Validation of client credentials against an authentication provider such as Active Directory, RADIUS, or SecurID Authentication Manager.
• Delegation of authentication to Web servers that are behind TMG Server, such as Exchange.

The first two phases are configured on the TMG Web listener while the third is configured on the specific publishing rules.

This fact can be used to use the same Web listener for different publishing rules with different authentication requirements.

Example

In the following example we will show how to publish two different internal Web sites on the same IP address and port (i.e. the same Web listener). One site will require authentication and the other will not. The technique used in this example can also be used for different paths instead of different sites.

Configuring the Web listener

The “trick” is to configure the Web listener to allow access to “All users”. This means that even though authentication is configured on the Web listener, it will not require users to authenticate (as “All users”, including non-authenticated users, are allowed). Here is an example of the Web listener configuration:

Configuring the Web publishing rules

Next we configure a Web publishing rule for the sites or paths that do not require authentication. We use the Web listener defined above and set the rule’s “Users” tab to apply to “All Users”:

Then we configure a Web publishing rule for the sites or paths that require authentication. We use the same Web listener as in the above rule, but here we set the rule’s “Users” tab to apply to “All Authenticated Users”:

We end up with two rules on the same listener (and the same IP address/port), one requiring authentication while the other doesn’t:

If a request matches the first, “No auth” rule, it will be allowed through without being prompted for authentication. However, if the request is matched to the second, “Auth” rule, it will be prompted for authentication and will only be allowed through if the authentication succeeds.

Authors:

Roman Golubchyck,  Senior Development Engineer.

Ori Yosefi, Senior Program Manager.

I’m happy to announce that Microsoft Forefront TMG 2010 (Standard Edition and Enterprise Edition) has passed Common Criteria Evaluation Assurance Level 4+ (EAL 4+).

The certification work has been performed by the Federal Office for Information Security (BSI), the Common Criteria certification body of the German government and TÜViT Evaluation Body for IT security which evaluates products worldwide according to the ITSEC and the Common Criteria (CC).

More information is available here.

Symptom

Consider that you have deployed Forefront TMG 2010 along with Exchange Edge and Forefront Protection for Exchange 2010 (FPE), and you have also enabled the E-Mail Policy feature in TMG.

In addition, you have configured from the TMG Management console some Content Filtering settings related to Exchange Edge.

The problem is that you notice that these settings are ignored. For instance:

You have defined a quarantine mailbox address based on spam confidence level (SCL) rating. Problem is that messages rated with a greater or equal SCL will never be sent to this mailbox.

The problem doesn’t happen for non Content Filtering settings like Allow/Block lists, Sender ID, etc …

Cause

The problem occurs because you have enabled the anti-spam filtering engine of FPE.

Actually when you deploy FPE, the anti-spam features that are built-in in Exchange Edge are disabled (see http://technet.microsoft.com/en-us/library/bb124739.aspx).

The anti-spam filtering of FPE can be enabled during installation of FPE, from FPE Management console or using the Set-FseSpamFiltering FPE cmdlet.

This means that if you open the Exchange Management Shell and execute the Get-TransportAgent cmdlet you will notice that the “Content Filter Agent” of Exchange is disabled. Instead, the “FSE Content Filter Agent” of FPE is enabled, as indicated in the screenshot below:

As a result, the settings mentioned above which belongs to Exchange are ignored because the “Content Filter Agent” of Exchange is disabled.

Solution and best practice

The best practice though is to enable the FPE anti-spam engine as it brings additional anti-spam filters like Cloudmark and Backscatter (see http://technet.microsoft.com/en-us/library/installingforefrontprotectiononastandaloneexchangeserver.aspx).

As explained above, the downside of enabling FPE anti-spam is that Exchange Content Filtering settings (configurable from the TMG MMC) that are performed thanks to the “Content Filter agent” are ignored.

If you need to use these Exchange filtering settings, you can perform the following task on each TMG array member:

• Configure Antispam Filtering manually in FSE Administration Console or Management Shell. Note: all subsequent anti-spam filtering settings changes should be done on each server separately as TMG doesn’t track these settings.
• Or disable Antispam Filtering (and restart MSExchangeTransport service)
• Or enable Content Filter Agent using Enable-TransportAgent cmdlet (and restart MSExchangeTransport service)

Note: there will be an improvement in the upcoming TMG Service Pack (SP2) that will notify the TMG administrator of the issue (when FPE anti-spam filtering is enabled and Exchange Content Filter agent is disabled).

Authors:

Eric Detoc, Escalation Engineer, Forefront TMG

Technical Reviewer:

Vadim Galperin, Development Engineer, Forefront TMG

Preface

Publishing SharePoint mobile for Windows Phone 7 with UAG is easy. However, it is possible to achieve similar results using TMG (or even other 3rd party reverse proxies)

In this post we will provide the high-level topology architecture used in order to access published SharePoint sites from WP7. This post also provides a step-by step demonstration using Forefront TMG.

High-Level Topology Architecture

The following is a flow description, explaining the roles of each of the above components:

1. John is the owner of a WP7 device and has just received an email from Lesley. The mail contains a link to Lesley’s page in the corporate SharePoint site. Note: the link sent by Lesley is the internal SharePoint URL, which is not publically accessible.

2. John clicks the link embedded within Lesley’s mail.

3. By clicking the link, John’s phone actually calls the SignURL.asp page that is hosted on the web server and is published by UAG. The full URL is: “https://<Reverse Proxy Address>/internalsite/SignUrl.asp?signURL=http://internal_server_name”.

4. The web server then replies to John’s phone with an echo of the SharePoint internal site name (copying the input parameter to the reply).

5. The reverse-proxy (UAG) intercepts the web server’s response and translates the SharePoint internal site name to an external name, one that John can access publically.

6. John’s phone is then redirected to Lesley’s external SharePoint site.

About SignURL.asp

This page is responsible for echoing the name of the internal server, so that it can be translated by the reverse-proxy, and thus informing the WP7 device of the public address of the internal server. This page must be placed under a virtual directory called internalsite (\internalsite\) because currently, WP7 makes a specific request to “https://<Reverse Proxy Address>/internalsite/SignUrl.asp”.

The request must also include a parameter named signURL in which the WP7 device sends the internal server name. An example of the full request made to the reverse proxy is: “https://<Reverse Proxy Address>/internalsite/SignUrl.asp?signURL=http://internal_server_name”.

The following is an example of a typical signurl.asp:

<%@LANGUAGE="VBSCRIPT" CODEPAGE="65001"%>

<%

Response.AddHeader "Cache-Control", "no-cache"

Response.AddHeader "Expires", "-1"

url = Server.HTMLEncode(Request.QueryString("SignURL"))

'the request does not contain url parameter

if Len (url) = 0 then

Response.status = 400

Response.write "SignURL parameter is not defined."

'url is not legal

elseif InStr(lcase(url),"http:\\") <> 0 and InStr(lcase(url),"https:\\") <> 0 then

Response.status = 400

Response.write "Invalid SignURL parameter structure."

else

Response.write url

end if

%>

This page relies on the reverse proxy engine to replace the internal_server_name with the proper public address name it was given, and let the WP7 device use this for the next request.

Other considerations and enhancements for hosting signURL.asp

It is possible to implement the signURL.asp in different languages, such as CGI (Common Gateway Interface). However, it is important to maintain identical functionality. When implementing the page in a different language, you must maintain the same URI, including the “asp” extension.

It is possible to host this page on a cluster, for fault tolerance and redundancy considerations.

It is possible to host this page on an existing web server. The virtual directory “internalsite” can be used to differentiate between the different websites.

Important things to note regarding the reverse-proxy and the client (WP7)

Reverse proxy:

1. The reverse proxy must publish the web server with the SignUrl.asp page using HTTPS without authentication.

2. The reverse proxy must be able to perform link translation for requests to the SignUrl.asp page.

3. The reverse proxy must be configured so it is able to translate the internal SharePoint URL to the external SharePoint URL

Client (WP7):

1. Windows Phone must be configured in the following way:

o The Forefront UAG address must be the reverse proxy address, which corresponds to the published web server with the SignUrl.asp page.

2. The client should trust the SSL certificate which is used to publish the SignUrl.asp page.

Using TMG instead of UAG

The following is a step-by-step configuration example, illustrating how TMG can replace UAG in the role of a reverse-proxy

1) Configure the Web listener for HTTPS traffic without authentication.

2) Publishing rule for SignUrl.asp page, which uses this web listener and has link translation enabled.

3) Publishing rule for SharePoint site with link translation enabled.

Notes:

· It is not required to publish the SharePoint site on the TMG machine. Instead, you can define a global mapping or local mapping for the SignUrl rule, which translates the internal SharePoint site name to its external name.

The purpose of this blog post is to provide you with some interesting information about the different kind of product updates the ISA/TMG Sustained Engineering (SE) team can release during the lifecycle of ISA Server or Forefront Threat Management Gateway.

First of all, we will make the distinction between a bug and a design change request (DCR):

• A bug is a problem in the code that leads the product to not behave as expected (unexpected crash for instance) or cause some regression compare to a previous release.
• A DCR is requested by customers to modify the way an existing feature works or to add a new feature in the product. For instance, the TFTP application filter introduced back in ISA Server 2006 is an example of the implementation of a DCR raised by one of our customers at that time.

Most of the times bugs and DCRs are raised by our customers through support cases. However they can also be raised internally by Microsoft IT, PSS and by the ISA/TMG Product team itself (developers, testers, Program Managers…).

The resolution of a bug (fix) or the implementation of a DCR can ship in different vehicles, which could be the following:

• A hotfix
• Available to customers on-demand via the PSS channel
• Code is based on last accumulative hotfix
• A public hotfix
• Available on Microsoft Update and/or Download Center
• Code is based on last accumulative rollup
• A security Update
• Available on Microsoft Update and Download Center
• Code is based on last accumulative rollup
• A Rollup
• Available each quarter (can be shorter) to customers via the PSS channel
• Code is based on the last accumulative rollup with the addition of all hotfixes approved since last rollup
• A Service Pack
• At least one Service Pack per year (according to a plan of release). Available on Microsoft Update and Download Center
• Code is based on last major release, include all rollups and approved hotfixes since last major release

Of course, all of the above is described in public KB articles.

Before releasing the official fix to a bug, the SE team generally releases an unofficial fix first (also known as “private” fix) except if the fix is planned to be part of next Service Pack or next major release. This unofficial fix is sent to the customer (who opened the support call) for testing. Once the customer approves the fix, the product team will package the fix in one of the shipping vehicles described above, based on the urgency of the bug.

It’s important to note that an unofficial fix does not undergo a full cycle of tests (only the official release does). It is intended to prove that the fix solves the bug. That’s why it is recommended to install and test an unofficial fix on a test environment, rather than the production environment.

Once the official fix is released (in a Rollup for instance), the customer must uninstall the unofficial fix first, then install the rollup.

In some very specific situations (when an issue is hard to troubleshoot), the SE team can provide an instrumented private update to the customer via PSS. This update is intended to provide detailed tracing information to the developer owning the bug in order to understand the root cause of an issue, and later on to design a fix solving the problem.

Another point to mention is that some official fixes (for specific issues) are disabled by default. A VB script must be then executed on the server or array in order to enable the fix. The script is always provided in the KB article describing the bug. Here is an example:

http://support.microsoft.com/kb/2445386/en-us

We hope you’ll find this information useful!

Author:

Eric Detoc, Escalation Engineer, Forefront TMG

Technical Reviewer:

Eyal Peri, Senior Program Manager, Forefront TMG Sustained Engineering team