Introduction

I have seen a number of cases where customers were installing TMG 2010 in a “hybrid” scenario. What I mean by this is that the EMS was part of the Domain but the Array Servers were in a workgroup. There are a couple of “gotchas” that I wanted to talk about today.

Assumptions

I am going to make a few of assumptions before I get started. First I am going to assume that you have already installed the TMG Enterprise Management Server (EMS) on a server that is a domain member. I am also going to assume that you have installed the TMG Array Member on a server that is in a workgroup. I recommend getting both of them to the latest Service Pack, Updates, and hotfixes before proceeding. You definitely want both the EMS and TMG Array server to be the same code level. Please refer here for version number information on TMG.

Certificates and Accounts

The first thing you want to do is request a Server Authentication certificate for your EMS. It needs to be issued to the Fully Qualified Domain Name (FQDN) of the EMS with the Private Key Exportable option checked. In my lab the EMS is called ems.fabrikam.com so I requested a Server Authentication certificate from my Certification Authority and installed it into the Certificate Store. Make sure you also export the .PFX file for the certificate, with the private key, and put the file somewhere handy on the EMS machine.

Next you will want to make sure that both the EMS and the TMG Array Server trust the CA that issued the Server Authentication certificate to the EMS. You can do this by importing the certificate for the CA into the Trusted Root Certificate Authorities branch in the Computer Store on each of those machines.

Another thing that is sometimes overlooked in the scenario is that mirrored accounts are needed on both the EMS and the TMG Array Server. For example, I just used the fabrikam/administrator account on the EMS and the local Administrator account on the TMG Array server. They both have the same password.

Keep in mind that if you have any firewalls that reside between the EMS and the TMG Array Servers you will initially want to allow ALL traffic between them. You can tweak this down later but it can cause you a lot of heartache with communications in TMG.

Create the New Array

On your EMS, open the TMG MMC, highlight the Arrays branch, and then on the far right-hand Tasks pane choose to Create New Array

Give your new array a name.

Type in the DNS name of the array.

Choose the Default Policy.

Click Next at the Array Policy Rule Types.

Complete the New Array Wizard.

TMG will create the new array and you should see that it was a success.

Apply this on the EMS.

Wait for the configuration changes to be saved.

In the TMG MMC on your EMS, there should now be a branch called Arrays. Below it should be the array that you just created.

Joining the New Array

Back on your TMG Array Server go into the TMG MMC and highlight the branch that says Forefront TMG (servername). On the far right-hand pane under the Tasks Tab, click Join Array.

You will see a welcome screen for the Join Array Wizard.

Under the Array Membership Type choose to “Join an array managed by an EMS Server”.

Give it the Fully Qualified Domain Name of your EMS. (Note: you will want to make sure name resolution is working properly on the TMG Array Server before you do this step).

The newly created array should come up as a choice.

Click Finish on the Completing the Join Array Wizard.

You should get a message that you successfully joined the array.

Give it a few minutes but you will probably notice that the configuration is not synching and you will get an error and a red X under the Configuration Status. The error reads “Forefront TMG Management cannot establish a connection with the Forefront TMG Computer.”

The Problem

So why isn’t the TMG Array Server able to communicate with the EMS? It seems like everything was set up correctly. TMG in a workgroup scenario relies on Authentication over SSL encrypted channel (LDAPS). That is the reason we requested the Server Authentication certificate for the EMS Server.

You can verify this by going into the MMC on your EMS, right-clicking on the top level branch of the array that you just created and choosing Properties.

Under the Configuration Storage branch the authentication type near the bottom should be set to “Authentication over SSL encrypted channel”

The problem is that the Server Authentication certificate was never bound to the ISASTGCTRL service.

You can verify this by creating a Certificates Snap-in MMC.

Choose to manage snap-in for a Service Account.

Select the Local Computer.

Select the ISASTGCTRL service and finish.

You should now see that there is not a certificate under the Personal branch.

Keep this open, you will refresh it in a few minutes.

To correct the certificate issue, download the TMG Cert Tool Pack from here.

Install the tool on the EMS but then move the ISACertTool.exe to the same directory where TMG is installed. Open an administrative command prompt and navigate to that directory. Run the command as explained below against the .PFX file you have for your EMS Server Authentication Certificate.

The syntax is listed in the DOC file that comes with it and is:

• ISACertTool /st file_name [/pswd password] [/keepcerts]

Where:

• /st file_name installs the exported certificate on the Configuration Storage server. File_name specifies the path and name of the exported .pfx certificate file.

• /pswd password specifies the password that may be required when installing the server certificate. It is only required if a password was specified during export of the certificate file.

• /keepcerts specifies that existing certificates should not be deleted. By default when you run ISACertTool.exe, all certificates in the ADAM_ISSTGCTRL local store are erased. To specify that existing certificates should not be deleted, specify the /keepcerts parameter.

After running this you should see a message that the Storage certificate installation was successful.

Go back to the Certificate Snap-in for the ISASTGCTRL Service and refresh the Personal Branch. You should now see the Server Authentication Certificate.

Now go back to Monitoring on your EMS and you should see that the TMG Server is successfully synching.

Conclusion

Setting TMG up where the EMS is in a domain and the TMG Array Servers are in a workgroup can be tricky. TMG in a workgroup relies on an SSL Encrypted channel and sometimes getting that to work correctly is not always straight-forward. In this article I have shown you a couple of the common pitfalls and how to correct them.

Author

Keith Abulton – Security Support Escalation Engineer, Microsoft CTS Forefront Security Edge Team

This post is about an issue that I came across, while working on a case recently.

SCENARIO:

The scenario was a simple website publishing through ISA server 2006. While accessing the HTTPS website from a client, we were seeing a Failed Connection attempt in the ISA logs. More specifically, the error message was pointing to Error code 13, “the data is invalid”.  First, we made sure the rule elements were configured correctly.  Next, while reproducing the issue, we collected an ISA Data Packager with the “Web Proxy and Web Publishing” template selected…and also collected a decrypted SSL Network trace from the ISA server.

DATA ANALYSIS:

From the decrypted SSL trace, we could identify that the ISA server returned a response code of 400 (BAD REQUEST). Upon further inspection of the HTTP headers and payload (the request was a POST), we could not find any issue which was obvious. So we looked further into the ISA Tracing file, which is a component level trace collected by the ISA Data Packager.

In that trace, the request seemed to contain a Multi-Line header which is not allowed by ISA server, by default. Hence it is rejecting the request made by the client.  The ISA tracing contained the following error:

ERROR:Multi line header - unsupported

We then revisited the decrypted SSL trace and observed that there was a MultiLine header found.

This trace example contains a multi-line header

In an attempt to reproduce the issue, we also used a web debugging tool to construct a similar HTTP POST request with the same payload and same headers and that worked as expected!  When we checked the network trace from our repro, we could see that the header was not sent as a “Multi-line”.

This trace example does not contain a multi-line header

The following byte codes (highlighted above) can be indicative of a multi-line header:

0d – Carriage Return

0a – New line (Line Feed)

09 – Horizontal tab

More information about the byte codes and their corresponding mappings can be found in RFC2616.

RESOLUTION:

In order to resolve the issue, we opened the registry editor on the problematic server and navigated to HKEY_LOCAL_MACHINE\SOFTWARE\MICROSOFT\RAT\Stingray\Debug\Web Filters

and added a DWORD value APPEND_CONTINUATION_LINES and set the value to 1, and then rebooted the server.

The issue no longer occurred. The client was able to access the site successfully.

The solution to this issue has been addressed in detail in the following KB article:

http://support.microsoft.com/kb/941293

This fix appends the continuation tab, thereby enabling the ISA server to successfully read the headers and process it further.

I hope this post would be helpful in resolving issues that you may face while publishing web applications through ISA/TMG.  But always remember not to make any changes to the registry unless you have isolated the root cause of the issue. Also, as a general best practice, take a backup of the registry before making modifications.

Author:

Karthik Divakaran

Security Support Engineer – MSD Security Team

Reviewers:

Junaid Jan

Security Support Escalation Engineer - MSD Security Team

Richard Barker

Sr. Security Support Escalation Engineer - MSD Security Team

As you may know, there is a known security vulnerability for the authentication method MS-CHAPv2.

The following TechNet article provides some detailed information about it:
Microsoft Security Advisory (2743314)

Unencapsulated MS-CHAP v2 Authentication Could Allow Information Disclosure

http://technet.microsoft.com/en-us/security/advisory/2743314

You may consider moving away from PPTP VPN connections which are configured to use this authentication method therefore.

But, what are the alternatives and how can you integrate these in TMG 2010 which is configured as a VPN Server?

As you can see, only VPN solutions that rely on PPTP in combination with MS-CHAP v2 as the sole
authentication method are vulnerable to this issue. If you need to rely on PPTP connections due to down level clients’ requirements, you may consider changing the authentication method for such PPTP connection to PEAP-MSCHAPv2. I will illustrate later how this can be achieved with TMG 2010.

First of all I would like to mention that you should consider using SSTP or L2TP/IPSec connections instead of PPTP.

Both types are also supported and integrated in TMG 2010.

Why you should deprecate PPTP?

PPTP uses Microsoft Point to Point encryption (MPPE). MPPE uses RC4 stream cipher for encryption. There is no method for authentication of the cipher text stream and therefore the cipher text is vulnerable to a bit-flipping attack.

You could use an authentication method like EAP-TLS, PEAP-TLS or PEAP-MSCHAPv2 for a PPTP connection to address the mentioned vulnerability in MS-CHAPv2. But the encryption method does not change by doing that.

So let’s take a look at L2TP/IPsec and SSTP.
The main advantage of Secure Socket Tunneling Protocol (SSTP) is that it is uses TCP port 443 as its server destination port since it relies on SSL which provides transport layer security. Therefore the SSTP packets can pass firewalls and proxies. Unfortunately there is no built-in SSTP support in down level or several 3^rd party operating systems. That’s why the affected administrators need to decide between PPTP and L2TP/IPsec for compatibility reasons.

L2TP/IPsec is well-known and has been widely used for many years. If you need some in-depth information, please refer to the following article:

http://technet.microsoft.com/de-de/library/cc771298(v=ws.10).aspx

Because of that fact that there is no need to reinvent the wheel, I would like to point to Deb Shinder who did a pretty good job writing a step-by-step guide on how to implement L2TP/IPsec in TMG 2010.

http://www.isaserver.org/tutorials/Checking-Out-TMG-2010-Virtual-Private-Network-Server-Part3.html

Though its quality, Microsoft does not adhere for third party information.

Before we look at the configuration of PEAP-MSCHAPv2, let me clarify that PEAP-MSCHAPv2 is NOT affected by the mentioned vulnerability, because the MSCHAPv2 information is only transmitted within a secure channel in PEAP phase 2.
The following article provides further information about the authentication method PEAP-MSCHAPv2 in general:

http://technet.microsoft.com/en-us/library/cc779326(v=WS.10).aspx
One more word before you read ahead. PPTP PEAP-MSCHAPv2 requires a certificate with Server Authentication EKU on a NPS Server from either your organizations PKI or a public CA.

Now let’s focus on how to implement PEAP-MSCHAPv2 in TMG. You may expect that TMG supports PEAP-MSCHAPv2 as valid authentication method, since it is some sort of EAP variant with improved security and because EAP can be enabled in TMG as authentication method. Unfortunately this is not 100% correct. If you choose EAP it in the Remote Access Policy it will cause the VPN connection to fail.

A Windows 7 client will report the following error:

Why does this happen?
The reason is that TMG configures the Routing and Remote Access Service (RRAS) and the Network Policy Service (NPS) which are included in the Windows Server operating system.

Doing this, TMG adds the following authentication method as a valid type to its local NPS configuration:
“Microsoft: Smart Card or other certificate” which is EAP-TLS, but neither PEAP-MSCHAPv2 or PEAP-TLS.

If you change the NPS configuration on the TMG Server, it will only provide a temporary workaround. The reason is that TMG will overwrite the NPS configuration once its service is restarted.

How you can solve that:

You will need to configure TMG to use Radius Authentication. This will result in outgoing Radius packets to a remote Radius/NPS Server. This additional NPS Server can be configured to enable PEAP support.

As next step you will need to define the remote RADIUS Server(s)

This is it from TMG perspective. After you have applied the configuration, you will need to configure the remote NPS Server.

After you have added the required NPS role on your remote server, open the NPS MMC.

First of all you will need to define and enable a RADIUS client there. This will ensure that incoming RADIUS packets from this source will not be dropped.

Next you will need to create a ‘Connection Request Policy’ and a ‘Network Policy’ in the remote NPS Server.

The following article explains the differences between ‘Connection Request Policies’ and ‘Network Policies’.

http://technet.microsoft.com/en-us/library/cc772279(v=WS.10).aspx

When installing NPS, you will have some policies already. Either you can edit them or create some new ones. We will create new ones for this example. Let’s start with a new ‘Connection Request Policy’ by using the wizard.

Then you will have to define conditions when this policy applies. You have lots of possibilities to define granular rules which match your needs. These possibilities are illustrated in the linked article above. To make things as easy as possible for this guide, we only use a Day-and-Time condition. Basically the condition always applies.

Leave the default settings on the next screen. You could forward the requests once more (Radius Proxy feature), but that does not make sense for this example.

If you are using Network Access Protection (NAP) for your VPN clients, you will need to use the setting “Override network policy authentication settings”. If you don’t use NAP, leave the default settings which I do for this straight-forward example.

Leave the settings on the next page to its defaults and click ‘Next’. After you have checked the summary, click on ‘Finish’. That’s it for the ‘Connection request policy’.

Let’s move on to the network policy.

Right click on ‘Network Policies’ and choose ‘New’.

In the next screen define a suitable policy condition which defines, when the network policies is about to be applied for a connection request. Again, you have plenty of options to use granular conditions. This does not only make sense, but may also be required due to your infrastructure needs. For example you may want to use different authentication methods, you may need to create different policies for VPN, 802.1x, etc.

To keep things simple, we use the Day and Time restriction once more. Again, using it in this way, it will always apply this rule.

Skip the next screen by clicking on ‘Next’. This will bring you to the configuration of the valid EAP types.

You will need PEAP-MSCHAPv2 for this scenario. Uncheck the ones which are not required and add PEAP-MSCHAPv2 by clicking on ‘Add’.

Click on ‘Ok’.

You screen should look like this:

Highlight the PEAP entry in the list and click ‘Edit’:

Make sure that you have chosen a valid and trusted certificate. Besides add EAP-MSCHAP v2 as inner method.

Click ‘Ok’, click ‘Next’ three times and then ‘Finish’ completing the configuration.

That’s it. I hope this guide helps you to understand if you need to change your VPN type and if needed, how to do it.

I am happy to answer related questions and read your feedback in the comments below.

Author:

Frank Hennemann

Microsoft CSS Forefront Security Edge Team

Technical Review:

Markus Sarcletti

Microsoft CSS Windows Networking

Sergio Medina here, and today I want to talk about a question we receive every now and then and explain what the solution is just in case some of you run into a similar issue. You could hit this issue when publishing an internal webserver that accepts only the username as a valid format. For example, when your web server is configured this way, if you try to log in directly to it using ‘DOMAIN\user’ as the format for the username you receive an error, however if you only include the ‘user’ part of the name in the user field, and provide a valid password of course, you log in successfully.

So let’s say you have an internal web server configured this way and you just configured your new web publishing rule in a Forefront Threat Management Gateway 2010 (TMG 2010) server/array. You used Form Based Authentication (FBA) as the authentication method in the web listener and HTTP Basic delegation in the web publishing rule.

Unfortunately, when you try to test your new publishing rule you find that it is not working as expected. If you analyze network traces between the TMG 2010 server and the webserver you discover that TMG 2010 is adding the NetBIOS domain name as part of the user name value:

Now we see the cause of the problem:

· The user introduces his/her username in the TMG form, as ‘user’
· TMG 2010 adds the NetBIOS domain name, sending the authentication in the format ‘DOMAIN\user’
· The webserver only accepts ‘user’ as a valid format so the request fails

So what could we do now?

a) Change webserver behavior, accepting also ‘DOMAIN\user’.

or

b) Change TMG behavior, sending only the username introduced by the user.

Solution a) is out of the scope of this blog article, so let’s see how to achieve solution b).

The trick here is to change the value of StripDomainFromCredentials, one of the internal properties of the web publishing rule. This property is not accessible using the Forefront TMG Management console so we need to change this is by using COM objects.

NOTE TheIFPCWebPublishingProperties2::StripDomainFromCredentials Property is documented here:http://msdn.microsoft.com/en-us/library/ff827099(v=vs.85).aspx

So with a simple script we could change this property for any web publishing rule. Here are the steps:

1. Copy the following lines into a new file named StripDomain.vbs on the TMG 2010 server:

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
' Microsoft Customer Technical Support
' Script for enabling StripDomainFromCredentials on a rule in TMG 2010  
' THIS CODE IS MADE AVAILABLE AS IS, WITHOUT WARRANTY OF ANY KIND. THE ENTIRE
' RISK OF THE USE OR THE RESULTS FROM THE USE OF THIS CODE REMAINS WITH THE
' USER. USE AND REDISTRIBUTION OF THIS CODE, WITH OR WITHOUT MODIFICATION, IS
' HEREBY PERMITTED.
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
Option Explicit  
Dim WebPublishingRuleName, newStripDomainFromCredentials  
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''  
' SET YOUR VALUES HERE  
' Rule name to change  
WebPublishingRuleName = "Rule name"  
' Set here custom values  
newStripDomainFromCredentials = True  
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''  
' Begin  
Dim Root, Array, Rules, Rule, intCompare  
Set Root = CreateObject("FPC.Root")  
Set Array = Root.GetContainingArray  
Set Rules = Array.ArrayPolicy.PolicyRules  
''''''''''''''''''''''''''''''''''  
' Look for the WebListener  
For Each Rule in Rules  
Wscript.Echo " Comparing Rule name |" & WebPublishingRuleName & "| with |" & Rule.Name & "|"  
intCompare = StrComp(WebPublishingRuleName, Rule.Name, vbTextCompare)  
If intCompare = 0 then  
Exit For  
End If  
Next  
Wscript.Echo  
Wscript.Echo "Found Rule with description: |" & Rule.Description & "|"  
''''''''''''''''''''''''''''''''''  
' Show values  
Wscript.Echo  
Wscript.Echo "***** CURRENT VALUES: "  
Wscript.Echo "** StripDomainFromCredentials = |" & Rule.WebPublishingProperties.StripDomainFromCredentials & "|"  
Wscript.Echo "***** NEW VALUES: "  
Wscript.Echo "** StripDomainFromCredentials = |" & newStripDomainFromCredentials & "|"  
''''''''''''''''''''''''''''''''''  
' Warning and ask to continue  
Dim strMessage  
WScript.Echo ' newline  
Wscript.Echo "Please check if the previous information is correct and you want to apply the changes"  
strMessage = "Press any key to continue or Ctrl+C to cancel"  
WScript.Echo ' newline  
WScript.StdOut.Write strMessage  
Do While Not WScript.StdIn.AtEndOfLine  
Input = WScript.StdIn.Read(1)  
Loop  
''''''''''''''''''''''''''''''''''  
' Set new values  
Rule.WebPublishingProperties.StripDomainFromCredentials = newStripDomainFromCredentials  
Wscript.Echo "***** CURRENT VALUES: "  
Wscript.Echo "** StripDomainFromCredentials = |" & Rule.WebPublishingProperties.StripDomainFromCredentials & "|"  
Rule.Save  
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

2. Modify the script, changing the value WebPublishingRuleName to the name of the rule you want to modify.

3. Open a command prompt with elevated privileges (Run as Administrator)

4. Run the script using: cscript /nologo StripDomain.vbs

5. Follow the steps in the script.

6. In an Enterprise environment, wait until the configuration changes are applied (Monitoring > Configuration > all servers in the array with Synced Status)

After this, you should see the TMG 2010 server sending only the username, thus resolving our issue:

If you need to undo the change, just edit the script, change the following variable to False and execute it again:

newStripDomainFromCredentials = False

One last thing to mention is that this default behavior has changed from ISA 2006 to TMG 2010 so you might have published your webserver successfully using ISA 2006 but found this new behavior when using TMG 2010.

Hope this helps,

Author: Sergio Medina | Senior Support Engineer | Microsoft CSS Forefront Edge Team

Technical Reviewer: Ashish Kapila | Support Escalation Engineer | Microsoft CSS Forefront Edge Team

Get the latest System Center news onFacebookandTwitter:

System Center All Up: http://blogs.technet.com/b/systemcenter/
System Center – Configuration Manager Support Team blog: http://blogs.technet.com/configurationmgr/
System Center – Data Protection Manager Team blog: http://blogs.technet.com/dpm/
System Center – Orchestrator Support Team blog: http://blogs.technet.com/b/orchestrator/
System Center – Operations Manager Team blog: http://blogs.technet.com/momteam/
System Center – Service Manager Team blog: http://blogs.technet.com/b/servicemanager
System Center – Virtual Machine Manager Team blog: http://blogs.technet.com/scvmm

Windows Intune: http://blogs.technet.com/b/windowsintune/
WSUS Support Team blog: http://blogs.technet.com/sus/
The AD RMS blog: http://blogs.technet.com/b/rmssupp/

The Forefront Endpoint Protection blog : http://blogs.technet.com/b/clientsecurity/
The Forefront Identity Manager blog : http://blogs.msdn.com/b/ms-identity-support/
The Forefront TMG blog: http://blogs.technet.com/b/isablog/
The Forefront UAG blog: http://blogs.technet.com/b/edgeaccessblog/

Hello Everyone! We’ve seen a few cases lately dealing with TMG servers sourcing outgoing packets with secondary IP addresses that have been added to the NICs. This could cause issues in communications between nodes or possibly other issues. One such example that I have seen come across is where a customer had a TMG server being utilized as an internal firewall behind a 3^rd party Edge firewall. Clients were utilizing the TMG server as their proxy server. When the http requests left the external interface of the TMG server the packets were sourced with a secondary IP address of the External NIC on the TMG instead of the primary address of that NIC. When the Edge firewall received the packets it dropped them because its rules were configured to only allow packets out when sourced with the primary IP address of the TMG’s external interface. This of course broke internet connectivity for all internal clients.

The question at hand is… “Why is the TMG server sourcing packets with a secondary address instead of the primary address of the NIC?”

The answer to that question deals with the differentiation between the Network Stack in Server 2008 and above and Server 2003 and below. Server 2003\XP and below were based off the Weak Host Model. Basically, in a Weak Host Model the primary address of the adapter with a route that most closely matches the target IP address is used as the Source IP Address.

In server 2008\Vista and above we re-architected the Network stack. It is based on the Strong Host Model. In the Strong Host Model the concept of a Primary IP Address doesn’t exist. To determine the IP address that is utilized it looks at the routing table to decide the proper NIC to utilize, then uses the process defined in RFC 3484 to choose the source IP for outbound packets. Here is a basic breakdown of how the windows implementation of RFC 3484 chooses an IP address:

Windows Source IP V4 address selection:

- Rule 1 Prefer same address (applies)

- Rule 2 Prefer appropriate scope (applies)

- Rule 3 Avoid deprecated addresses (applies)

- Rule 4 - Prefer home addresses - does not apply to IP v4

- Rule 5 Prefer outgoing Interfaces (applies)

- Rule 6 Prefer matching label - does not apply to IP v4

- Rule 7 Prefer public addresses - does not apply to IP v4

- Rule 8a: Use longest matching prefix with the next hop IP address. (not in RFC!)

"If CommonPrefixLen(SA, D) > CommonPrefixLen(SB, D), then prefer SA. Similarly, if CommonPrefixLen(SB, D) > CommonPrefixLen(SA, D), then prefer SB."

*This says that the IP with the most high order bits that match the destination of the next hop will be used.

Note: Rule 8 - Use longest matching Prefix is similar to rule 8a except the match is with the destination IP address rather than the next hop IP address.

For example, use the following addresses as an example of choosing the longest matching prefix:

TMG Servers External IP Addresses:

192.168.1.14/24
&
192.168.1.68/24

Default Gateway:

192.168.1.127

Convert these addresses into binary:

192.168.1.14   = 11000000 10101000 00000001 00001110
192.168.1.68   = 11000000 10101000 00000001 01000100
192.168.1.127 = 11000000 10101000 00000001 01111111

The 192.168.1.68 address has more matching high order bits with the gateway address 192.168.1.127. This would cause the server to utilize what was originally defined as the “secondary” as the Source IP address of outgoing packets.

*For more information on RFC 3484 please refer to the following link: http://www.ietf.org/rfc/rfc3484.txt . Please note that IPv6 is referenced in RFC. Windows utilize the same process for IPv4 sourcing.

You can also review the following TechNet article for supported document details on the above information:

The functionality for source IP address selection in Windows Server 2008 and in Windows Vista differs from the corresponding functionality in earlier versions of Windows
http://support.microsoft.com/kb/969029

So now we know why your TMG server may be sourcing your packets with what you call your “Secondary IP Address”. It isn’t TMG at all. It is the default behavior of the server itself. Your server version is Server 2008 or above. The question is…

“Can I configure my Server 2008 or above in a way that it will only utilize the first IP address as a Source address?”

The answer to that is YES! There is actually a Netsh command that can be utilized to add IP addresses.  In that command you use the “SkipAsSource” flag and it will no longer use the IP address you are adding as a Source IP Address. This means that you will have to temporarily remove the IP Address you are having the issues with then re-add them utilizing the Netsh command (This means you will have to have a maintenance window!). Here are examples of the command lines you will use:

Server 2008:
Netsh int ipv4 add address <Interface Name> <ip address> <subnet mask> skipassource=true

Server 2008 R2:

Netsh int ipv4 add address <Interface Name> <ip address> skipassource=true

* For details and prerequisites to utilize these commands please refer to the following articles:

All IP addresses are registered on the DNS servers when the IP addresses are assigned to one network adapter on a computer that is running Windows Server 2008 SP2 or Windows Vista SP2
http://support.microsoft.com/default.aspx?scid=kb;EN-US;975808

IP addresses are still registered on the DNS servers even if the IP addresses are not used for outgoing traffic on a computer that is running Windows 7 or Windows Server 2008 R2
http://support.microsoft.com/default.aspx?scid=kb;en-US;2386184

Keep in mind that I gave only one specific example where this may be causing an issue.  There may be other problems you run into where the Netsh entry I listed may help you out.  No telling… it may not even be on your TMG servers.  Maybe you see the issue on your UAG servers, or any other server for that fact. 

I hope the information provided helps out!

Author

Brett Crane - Sr Security Support Escalation Engineer, Microsoft CTS Forefront Security Edge Team

Reviewer

Richard Barker - Sr Security Support Escalation Engineer, Microsoft CTS Forefront Security Edge Team

This post is, once again, about an issue I worked on few days back.  Before I start discussing the issue, and how I resolved it, I would like to outline the objective of this post.

The objective of this post is to make TMG administrators aware of issues like this; and what can be done to resolve them. Discovering the root cause of this issue required a User Mode Dump analysis.

Performing a User Mode Dump analysis requires “Symbol” files (which are private). My goal is not to provide specific instruction on User Mode Dump analysis, but instead to show what kind of information can be gathered, and how it can be used, to help troubleshoot boot-time “service issues” on a TMG server.

For those that are not familiar with dump analysis terms like process, threads and its stack, I will elaborate further as I explain the steps.

Issue:

TMG server admin was rebooting the server and at the time of reboot TMG services were hanging and were not starting. A similar issue was reported pre TMG sp2 but it was fixed post sp2. In this scenario TMG was updated to latest build i.e. TMG sp2 RU2.

Troubleshooting:

Some background: It should be noted that quite a bit of troubleshooting had taken place prior to my involvement in the case. This includes the steps in the following Knowledge Base article:

Forefront Threat Management Gateway 2010 services do not start as expected when the FTMG 2010 servers are in a workgroup array

During startup, the following System Event was logged…

_____________________________________________________________________________________________

Log Name:      System
Source:        Service Control Manager
Date:          09/11/2012 17:42:30
Event ID:      7022
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      server1
Description:
The Microsoft Forefront TMG Firewall service hung on starting.

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
    <EventID Qualifiers="49152">7022</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2012-11-09T17:42:30.378163900Z" />
    <EventRecordID>344470</EventRecordID>
    <Correlation />
    <Execution ProcessID="716" ThreadID="720" />
    <Channel>System</Channel>
    <Computer>server1</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="param1">Microsoft Forefront TMG Firewall</Data>
  </EventData>
</Event>

_____________________________________________________________________________________________

Data collection:

During the course of troubleshooting we collected a User Mode Dump while reproducing the issue.

User mode dumps collection reference: http://msdn.microsoft.com/en-us/library/ff420662.aspx

Data analysis:

Note: The approach taken in this post is very similar to guidelines given in the following link about debugging a deadlock as we were in a scenario similar to a deadlock: http://msdn.microsoft.com/en-us/library/windows/hardware/ff540592(v=vs.85).aspx

In the dump, I found following critical section was locked :

Note: For more information about critical section and locked critical section, please refer to: http://msdn.microsoft.com/en-us/library/windows/hardware/ff541979(v=vs.85).aspx

Then I located the owning thread of this locked critical section. In following snapshot we can see the stack of this thread. The stack is read from bottom to top. From this call stack it appears that wspsrv (firewall service) is trying to load a filter called XSISAPI. It appears TMG has deferred its filters’ startup until this filter (i.e. XSISAPI)is loaded.

I then checked the module for this filter (i.e. XSISAPI) and found that it’s a filter called “Afaria” from Sybase.

Solution:

We configured the XSISAPI filter service to delayed start. After this change, the TMG services started normally after reboot.

Author:

Suraj Singh:

Security Support Escalation Engineer - MSD Security Team

Reviewer:

Richard Barker

Sr. Security Support Escalation Engineer - MSD Security Team

In these days we received a considerable number of support requests asking for more info about SSL/TLS Renegotiation and the risk it introduces of being exposed to DoS attacks and malicious code injections.

The requests in object were focused on ISA/TMG products, considering they are used as reverse proxy for web publishing purposes, but the below considerations can be considered valid for every kind of Windows server/client supporting SSL/TLS connections.

First, what is exactly SSL/TLS Renegotiation?

"TLS [as defined in RFC 5246] allows either the client or the server to initiate a renegotiation -- a new handshake that establishes new cryptographic parameters. Unfortunately, although the new handshake is carried out using the cryptographic parameters established by the original handshake, there is no cryptographic binding between the two. This creates the opportunity for an attack in which the attacker who can intercept a client's transport layer connection can inject traffic of his own as a prefix to the client's interaction with the server"

The above definition is taken from RFC 5746.

The following is a graphic representation of a basic SSL/TLS Handshake:

Under certain circumstances, the client could be asking the server a renegotiation of the SSL/TLS parameters using the same TCP socket:

If the SSL/TLS is not secure (as per RFC 5746 recommendations) a MITM could use the renegotiation to send the server malicious data, pretending to be "good" user.

What chances do we have to mitigate this issue?

You should make sure, that the following security hotfix is installed:

http://support.microsoft.com/kb/980436

This fix is making the system compliant with RFC 5746, mitigating the risk of malicious data injection.

To provide backward compatibility, this security update works in the following modes: STRICT and COMPATIBLE

Compatible mode

o If this security update is applied to the server, and the server is in compatible mode, the server allows all clients to set up and renegotiate Transport Layer Security (TLS) sessions. This occurs whether the clients are updated or are not updated by using this security update.

o Similarly, if this security update is applied to the client, and the client is in compatible mode, the client can set up and renegotiate TLS sessions with all the servers for which this security update is applied or is not applied.

Strict mode

o If this security update is applied to the server, and the server is in strict mode, the server allows only those clients to which this security update is applied to set up and renegotiate TLS sessions. The server does not allow the clients to which this security update is not applied to set up the TLS session. In this case, the server terminates such requests from the clients.

Similarly, if this security update is applied to the client, and the client is in strict mode, the client can set up and renegotiate TLS sessions with all the servers for which this security update is applied. The clients cannot set up TLS sessions at all with servers for which this security update is not applied. The client cannot move ahead with a TLS negotiation attempt with such servers.

By default, this security update enables the TLS or Secure Sockets Layer (SSL) client or server to stay in compatible mode. An administrator can use the AllowInsecureRenegoClients and the AllowInsecureRenegoServers entry DWORD values in the following registry path to enable strict mode on the client or on the server:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL

The following table shows how these DWORD values can be used:

Malicious data injection is not the only problem related to Renegotiation: it can be in fact used to perform DoS attacks against a server.

When a new SSL/TLS connection is being negotiated, the server will typically spend significantly more CPU resources than the client. A malicious user, leveraging the Renegotiation, could be able to enhance the server's CPU usage causing DoS.

In order to have a better mitigation for both malicious data injection and DoS attacks, the best option would be to reject the client-initiated SSL/TLS renegotiation at all.

The following Microsoft Security Advisory explains how:

http://support.microsoft.com/kb/977377/en-us

As reported in the article, the behavior can be modified by changing the value of the following registry key:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\DisableRenegoOnServer

Notes

· If the DisableRenegoOnClient subkey is present and has any nonzero value:

o The client will not initiate renegotiation.

o The client will not respond to renegotiation.

· If the DisableRenegoOnClient subkey is missing or is present and has a zero value:

o The client will initiate renegotiation.

o The client will respond to renegotiation.

· If the DisableRenegoOnServer subkey is present and has any nonzero value:

o Server initiated renegotiation is not allowed.

o The server will not respond to renegotiation requests from the client.

· If the DisableRenegoOnServer subkey is missing or is present and has a zero value:

o Server initiated renegotiation is allowed.

o The server will respond to renegotiation requests from the client.

Of course, this may have an impact on the use of specific applications requiring SSL/TLS renegotiation feature.

The KB article underlines the following:

o After you install this security update, you cannot use the legacy provisioning service parameter (–UseLegacyProvisioningService) when you create a federation trust with the Microsoft Federation Gateway. The security update will prevent the federation trust from working correctly. This problem will occur if you install this security update on a computer that is running Exchange Server 2010 or Exchange Server 2010 Service Pack 1 before you have created a federation trust. To avoid this problem, you must create the federation trust before you install this security update. 
For more information about how to create a federation trust by using the –UseLegacyProvisioningService parameter, visit the following Microsoft webpage:
Create a Federation Trust

o When you install this update on a computer that has the Microsoft Online Single Sign-In services client installed, you may experience the following issues:

· The Sign In client cannot obtain user configuration information. This only affects new users who are running the Sign In client for the first time. The Sign In client cannot obtain information to configure Outlook. If the Sign In client has already run and configured the applications, there are no additional issues with the Sign In client.

· Outlook users cannot see free/busy information. Therefore, this update also affects existing Outlook users.
To resolve this problem, set the DisableRenegoOnClient registry entry to a value of 0 (zero), and then restart the computer.

o This update disables TLS/SSL renegotiation, common protocol functionality that is required for specific applications. This may cause this software to no longer function as expected. If any side effects are experienced, customers should uninstall the workaround to resolve the issue.
The following software has been tested by Microsoft and that has been found to experience problems when you install this update:

· Windows 7 DirectAccess: The IP HTTPS interface will not function.

· Exchange ActiveSync: Does not function when it uses certificate client authentication.

· Internet Information Services (IIS): In certain configurations, IIS using certificate client authentication, including certificate mapping scenarios, will be affected. Site-wide client certificate authentication will not be affected and will continue to function.

· Internet Explorer: When you browse Web sites that require client certificate authentication, but not site-wide client certificate authentication, you may not successfully be able to connect.

Of course, it's not possible to predict the implications of disabling client-initiated renegotiation for various applications: this solution should be appropriately tested in each specific environment.

From a security point of view, though, this is the recommended way to mitigate all the above described problems.

Hope this can help!

Author: 

Daniele Gaiulli

Support Engineer – Microsoft Forefront Edge Security Team

Reviewer:

Philipp Sand

Sr. Support Escalation Engineer – Microsoft Forefront Edge Security Team

We are happy to announce the availability of Rollup 4 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 2 (SP2). TMG SP2 Rollup 4 is available for download here: Rollup 4 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 2

Please see KB Article ID: 2870877 for details of the fixes included in this rollup. The Build Number for this update is: 7.0.9193.601

To install this update, you must be running Forefront Threat Management Gateway 2010 Service Pack 2.

For more information about Forefront Threat Management Gateway 2010 SP2, please see the following
Microsoft website:

Download information for Forefront TMG 2010 SP2

Thank you,

Forefront TMG Team

Microsoft’s own Junaid Jan (Security Support Escalation Engineer - Forefront Edge Team) recently wrote a great article in our TechNet Wiki about an issue where an enterprise node is incorrectly added in the Forefront TMG 2010 management console after you run a repair on Forefront TMG 2010 SP1 Update 1. When this happens, you won't be able to add that server to an array because it thinks it's already part of an array. To fix it we need to install TMG 2010 SP1 Update 1 Rollup 3 and run a script as well.

For all the details, including a link to the update and his script see the following:

An enterprise node is incorrectly added in the Forefront TMG 2010 MMC after you run repair on Forefront TMG 2010 SP1 Update 1 (http://social.technet.microsoft.com/wiki/contents/articles/13053.an-enterprise-node-is-incorrectly-added-in-the-forefront-tmg-2010-mmc-after-you-run-repair-on-forefront-tmg-2010-sp1-update-1.aspx)

J.C. Hornbeck| Knowledge Engineer | Management and Security Division

Get the latest System Center news onFacebookandTwitter:

App-V Team blog: http://blogs.technet.com/appv/
ConfigMgr Support Team blog: http://blogs.technet.com/configurationmgr/
DPM Team blog: http://blogs.technet.com/dpm/
MED-V Team blog: http://blogs.technet.com/medv/
Orchestrator Support Team blog: http://blogs.technet.com/b/orchestrator/
Operations Manager Team blog: http://blogs.technet.com/momteam/
SCVMM Team blog: http://blogs.technet.com/scvmm
Server App-V Team blog: http://blogs.technet.com/b/serverappv
Service Manager Team blog: http://blogs.technet.com/b/servicemanager
System Center Essentials Team blog: http://blogs.technet.com/b/systemcenteressentials
WSUS Support Team blog: http://blogs.technet.com/sus/

The Forefront Server Protection blog: http://blogs.technet.com/b/fss/
The Forefront Endpoint Security blog : http://blogs.technet.com/b/clientsecurity/
The Forefront Identity Manager blog : http://blogs.msdn.com/b/ms-identity-support/
The Forefront TMG blog: http://blogs.technet.com/b/isablog/
The Forefront UAG blog: http://blogs.technet.com/b/edgeaccessblog/

Problem

When publishing Exchange 2010 “Outlook Anywhere” via TMG 2010, you may find that some of your external Outlook users may intermittently experience issues sending email. They may report, when sending a new email, that the email may get “stuck” in the Outbox folder. The users may find that the email will be sent after a random number of minutes…or not at all.  Forcing a Send and Receive does not help. However, they may find that if they close and restart the Outlook client, the items are then sent.

The difficulty in troubleshooting this problem is that none of the endpoints in question will log any relative error messages. Neither the Outlook client, TMG nor the Exchange CAS server log any events or errors that appear relative to the issue.

Explanation

This turns out to be a timing issue which can result in ‘orphaned’ TCP connections. The Outlook client has a default RPC timeout of 12 minutes. The server to client default RPC timeout is 15 minutes.

In a publishing scenario that allows access from external clients, it’s not unusual to have a number of different network devices between the Outlook client and the internal Exchange CAS servers.  If the TCP connection timeout of one or more of these devices is sufficiently low enough, the TCP connection may be dropped by the device, causing the RPC connections between the Outlook client and the Exchange CAS server to drop. In our scenario, the device we’re interested in is TMG.

A TMG SP2 server has a default TCP keepalive value of 5 minutes. Therefore, it’s possible that TMG may drop the RPC connection from an ‘idle’ Outlook Anywhere client.

More information

The registry value that controls the Exchanges RPC Proxy connection timeout is:

HKLM\Software\Policies\Microsoft\Windows NT\Rpc\MinimumConnectionTimeout

The TMG servers’ registry value that controls TCP/IP keepalive time is:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\KeepAliveTime

NOTE: The value of MinimumConnectionTimeout is specified in seconds and the value of KeepAliveTime is specified in milliseconds.

Resolution

Decrease the Exchange CAS servers’ RPC Proxy timeout to be less than the TMG servers’ TCP keepalive time. As the default TCP keepalive value on TMG is 5 minutes, you can configure the CAS servers’ RPC Proxy timeout to 3 minutes (180 seconds) as follows:

HKLM\Software\Policies\Microsoft\Windows NT\Rpc\MinimumConnectionTimeout DWORD 0x000000b4 (180)

NOTE: The MinimumConnectionTimeout registry value does not exist by default. You’ll need to create it if it doesn’t exist in this location. Also note that adding and/or editing this registry value will require a reboot of the Exchange CAS server.

Don’t forget to check other devices on the network and make sure they do not have TCP timeout settings that might be lower than your newly configured RPC Proxy MinimumConnectionTimeout values.

Author

Richard Barker

Sr. Security Support Escalation Engineer

Microsoft CSS Forefront Edge Team

Today, as a result of our effort to better align security and protection solutions with the workloads and applications they protect, Microsoft is announcing changes to the roadmaps of some of the security solutions made available under the Forefront brand.

1. As part of this effort, the next release of Forefront Online Protection for Exchange, which has long been part of the Office 365 solution, will be named Exchange Online Protection.
2. In response to customer demand, we are adding basic antimalware protection to Exchange Server 2013. This protection can be easily turned off, replaced, or paired with other services (like Exchange Online Protection) to provide a layered defense.
3. We are discontinuing any further releases of the following Forefront-branded solutions:

• Forefront Protection 2010 for Exchange Server (FPE)
• Forefront Protection 2010 for SharePoint (FPSP)
• Forefront Security for Office Communications Server (FSOCS)
• Forefront Threat Management Gateway 2010 (TMG)
• Forefront Threat Management Gateway Web Protection Services (TMG WPS)

For all the details please see the following: http://blogs.technet.com/b/server-cloud/archive/2012/09/12/important-changes-to-forefront-product-roadmaps.aspx

J.C. Hornbeck| Knowledge Engineer | Management and Security Division

Get the latest System Center news onFacebookandTwitter:

App-V Team blog: http://blogs.technet.com/appv/
ConfigMgr Support Team blog: http://blogs.technet.com/configurationmgr/
DPM Team blog: http://blogs.technet.com/dpm/
MED-V Team blog: http://blogs.technet.com/medv/
Orchestrator Support Team blog: http://blogs.technet.com/b/orchestrator/
Operations Manager Team blog: http://blogs.technet.com/momteam/
SCVMM Team blog: http://blogs.technet.com/scvmm
Server App-V Team blog: http://blogs.technet.com/b/serverappv
Service Manager Team blog: http://blogs.technet.com/b/servicemanager
System Center Essentials Team blog: http://blogs.technet.com/b/systemcenteressentials
WSUS Support Team blog: http://blogs.technet.com/sus/

The Forefront Server Protection blog: http://blogs.technet.com/b/fss/
The Forefront Endpoint Security blog : http://blogs.technet.com/b/clientsecurity/
The Forefront Identity Manager blog : http://blogs.msdn.com/b/ms-identity- support/
The Forefront TMG blog: http://blogs.technet.com/b/isablog/
The Forefront UAG blog: http://blogs.technet.com/b/edgeaccessblog/

One of the problems causing TMG to log to LLQ instead of the database is the presence of orphaned databases in the local SQL Server instance.
In other words you may have some databases that are registered on the local SQL Server but the corresponding .mdf and .ldf files are missing from the disk. This may occur if the files were manually deleted, the volume with the files are no longer available or for other reasons.

It’s also important to note that this may happen when logging is configured for either the local database or remote SQL database. This is because TMG still checks the health of the local instance even if you log to a remote database.

When this problem occurs you may see something like this:

To understand if this is the case you will need to check the logs of the local SQL Server instance that are located by default in C:\Program Files\Microsoft SQL Server\MSSQL10.MSFW\MSSQL\Log while the databases themselves are by default in 'C:\Program Files\Microsoft Forefront Threat Management Gateway\Logs\.

Open the file ERRORLOG from the logs folder and search for lines that looks like the following:

2012-09-05 10:44:52.01 spid54 Starting up database 'ISALOG_20120831_FWS_000'.
2012-09-05 10:44:52.02 spid54 Error: 17204, Severity: 16, State: 1.
2012-09-05 10:44:52.02 spid54 FCB::Open failed: Could not open file C:\Program Files\Microsoft Forefront Threat Management Gateway\Logs\ISALOG_20120831_FWS_000.mdf for file number 1. OS error: 2(failed to retrieve text for this error. Reason: 15100).
2012-09-05 10:44:52.15 spid54 Error: 17207, Severity: 16, State: 1.
2012-09-05 10:44:52.15 spid54 FileMgr::StartLogFiles: Operating system error 2(failed to retrieve text for this error. Reason: 15105) occurred while creating or opening file 'C:\Program Files\Microsoft Forefront Threat Management Gateway\Logs\ISALOG_20120831_FWS_000.ldf'. Diagnose and correct the operating system error, and retry the operation.

Next thing you should check is what happened to the missing files.

If you changed the log location to another volume and this volume is currently unavailable then try to get the volume back online if possible.

If there is no way to get those files back you should proceed by removing the database registrations from the local master database. You can identify the names of the orphaned databases by running this command at an elevated command prompt:

OSQL -E -S .\MSFW -Q "select name from sysdatabases where name like '%isalog%'"

Compare the names from the output of the above command with the database files in the log file directory. Once you have identified the names of the missing databases you should prepare a text file with the commands to drop each missing database. It should look something like this:

drop database ISALOG_20120831_FWS_000
go
drop database ISALOG_20120831_WEB_000
go
drop database ISALOG_20120901_FWS_000
go
drop database ISALOG_20120901_WEB_000
go

Save this file, for example, as c:\DropDB.sql

Then from an elevated command prompt execute this command:

OSQL -E -S .\MSFW -i c:\DropDB.sql

Now restart the “Microsoft Forefront TMG Firewall” service and then check back the Log Status, you should see the current status no longer as “Disconnected” but as “Queue in use”. If you click Refresh you should also see the LogQueue(KB) decreasing.

Depending on how long the issues lasted and the amount of data being logged by your server, it may take few minutes or a few days for the queue data to be processed.

Once this operation completes you should see the status as Ready again.

Author:
Gianni Bragante
Support Engineer - Microsoft CSS Forefront Security Edge Team

Reviewer:
Lars Bentzen
Escalation Engineer - Microsoft CSS Forefront Security Edge Team

It can happen while installing Forefront TMG 2010 or during a repair that we hit the following error:

To this error is also normally linked to the ISA managed control service not starting correctly and errors as the following in the application event viewer:

On the other hand it is also possible to hit the above error “setup failed while registering Forefront TMG managed performance monitor” as a consequence of a troubleshooting to fix a “down” of the ISA managed control service as it was in our case.

Unfortunately both errors are not really self-explicative and especially if the TMG box has installed on top of it the Edge role of exchange and the Forefront protection for exchange the very first temptation is to remove everything and reinstall all. This can mainly waste a lot of our time without any final benefit if we are under the condition of our blog post. In fact the above errors are not linked to any specific wrong condition of one of the component installed under TMG (an input string error as per above screenshot could mislead us).

As per our case the issue is very probably linked to a possible corruption in the performance monitor counters of the OS.

The solution in our case was rather straightforward and at the same time simple. For some reasons we got a corruption in the performance monitor files. This can be because of many different reasons but the most probable one as per our case is a power loss/blue screen.

To fix this apparently very bad issue it is enough to run the following command after moving under windows\system32:

lodctr /R

The command updates registry values related to performance counters and the option /R rebuilds the performance registry strings and info from scratch based on the current registry settings and backup INI files.

That’s all. We have successfully fixed the corruption error linked to the performance counters that had as back effect to prevent the correct startup of the ISA managed control service.

We are now able to bring the ISA managed control service up and running normally.

Author
Andrea Vescovo
Support Engineer
Microsoft CSS Forefront Edge Team

Technical Reviewer
Philipp Sand
Support Escalation Engineer
Microsoft CSS Forefront Edge Team

Introduction

A much needed feature was added in Service Pack 2 for Forefront TMG 2010. This great new feature gives you the ability to lock accounts on TMG at the local level before accounts are actually locked out in the domain. The account lockout feature, when used properly, will prevent TMG from trying to authenticate a user to a Domain Controller after the defined number of bad passwords has been attempted.

In one of my previous blogs I talked about scenarios where TMG is being used as a reverse proxy and the Account Lockout Threshold has been set in the AD domain. Often times, when companies require their users to change passwords at a given interval, devices will end up with a bad password stored on them. Devices may use the old password for Exchange ActiveSync over and over again until the domain account has been locked out.  This can cause a lot of frustration for IT departments that are trying to track down the source of the lockouts and also having to frequently unlock accounts.

To enable this new feature, install SP2 which you can get here.  The details for the new lockout feature can be found here.

The new feature, however, is not automatically enabled after installing the Service Pack and cannot be modified using the TMG GUI.  The account lockout feature can only be modified through the Forefront TMG Com Object Model.  Fortunately for us there are examples available out there on how to do this using PowerShell. One such example has been provided to us by Jan Egil Ring in the Microsoft Script Center and it is located here.

There are a couple of important things to keep in mind when using this feature:

1.)    Only Web Listeners configured to use Forms Based Authentication (FBA) can be configured to use the new feature.

2.)    The value you choose for the AccountLockoutThreshold variable is set on TMG on a “per server” basis and must be set on each TMG server
individually.

Scenario

Let’s look at a specific example to help you understand how the feature works and the implications of using it. In our example I am the administrator for a domain called Fabrikam.com and I have decided that I want to set the number of bad password attempts allowed in my Fabrikam Domain to 5. I would normally do this by setting the Account Lockout Threshold value in my Default Domain Policy using the Group Policy Editor. I also require that my users change their passwords every 30 days.

Now let’s assume also that I have 3 TMG Servers (TMG 1, TMG2, and TMG3) that sit on the edge of my network. The 3 TMG Servers are load balanced using Microsoft’s built in Network Load Balancing (NLB). These servers are used only for reverse proxy (Outlook Web Access and Exchange ActiveSync) so that my users can retrieve their email while on the road or at home. I have 2 Publishing Rules on TMG, 1 is for OWA and the other is for Exchange ActiveSync. They both share the same Web Listener which uses FBA Authentication.

Everything is going fine until one day we start getting reports that accounts are being locked out. It appears that some of the Executives have devices managed by their administrative assistants. My domain password policy requires users to change their passwords every 30 days. After some initial investigation it is found out that these devices were storing bad passwords and causing the account lockouts.

I have just learned of the new account lockout feature on TMG and I follow the links given earlier and configure it so that the AccountLockoutThreshold value is set to 3 on each one of my TMG Servers. My thinking is that the local value of 3 that I am using on TMG is lower than my Domain value of 5 so I should be okay.  If TMG does its job correctly, it will stop trying to authenticate bad password attempts before the devices lock the domain accounts out.

Even after setting the lockout feature on TMG I am still getting reports of accounts being locked out in my domain. What did I do wrong? Keep in mind something that I pointed out earlier which is that this value is “per TMG Server”. In my example I have 3 TMG servers that are being load balanced. Devices, such as phones, that are used for Exchange ActiveSync will sometimes get a different IP address every couple of minutes for their carrier. Because TMG is using NLB, this device may be serviced by any 1 of the 3 TMG Servers (depending on a few factors).

Let’s say that the CIO of Fabrikam, John Smith, has a phone which is carried by his admin assistant and has an IP addresses provisioned to it by the carrier that ends in 10. John recently changed his domain password but it was not changed on the phone that his administrative assistant carries. The phone is now attempting to retrieve his email using the old (incorrect) password on TMG1 since it has been load balanced to it. TMG1 tries to authenticate John Smith to a Domain Controller 3 times using the bad password. On the fourth attempt, TMG1 no longer tries to authenticate to a DC. So far so good right? TMG has done its job and the account is not locked out. Let’s further assume that John Smiths phone hits another cell tower and is now provisioned an IP address that ends in 11. Because of the way NLB works, the phone tries to use TMG2 to retrieve John's email.  Again it tries 3 times using the bad password.  Since the AccountLockoutThreshold value is “per server”, TMG2 also tries to authenticate the account to a Domain Controller.  From the perspective of the Domain Controller, a bad password attempt has been tried 6 times so the account is locked out.

Conclusion

As you can see from the example above you will need to tweak the settings on the TMG server and possibly even on your Domain Password Policy to get the desired results from this new feature. This will all depend on your individual environment and certain factors such as how many TMG servers you are using, how they are load balanced, and what types of devices are potentially trying bad passwords.

Author

Keith Abluton – Security Support Escalation Engineer, Microsoft CTS Forefront Security Edge Team

Reviewer

Richard Barker – Sr. Security Support Escalation Engineer, Microsoft CTS Forefront Security Edge Team

Introduction

I have seen a number of cases where customers were installing TMG 2010 in a “hybrid” scenario. What I mean by this is that the EMS was part of the Domain but the Array Servers were in a workgroup. There are a couple of “gotchas” that I wanted to talk about today.

Assumptions

I am going to make a few of assumptions before I get started. First I am going to assume that you have already installed the TMG Enterprise Management Server (EMS) on a server that is a domain member. I am also going to assume that you have installed the TMG Array Member on a server that is in a workgroup. I recommend getting both of them to the latest Service Pack, Updates, and hotfixes before proceeding. You definitely want both the EMS and TMG Array server to be the same code level. Please refer here for version number information on TMG.

Certificates and Accounts

The first thing you want to do is request a Server Authentication certificate for your EMS. It needs to be issued to the Fully Qualified Domain Name (FQDN) of the EMS with the Private Key Exportable option checked. In my lab the EMS is called ems.fabrikam.com so I requested a Server Authentication certificate from my Certification Authority and installed it into the Certificate Store. Make sure you also export the .PFX file for the certificate, with the private key, and put the file somewhere handy on the EMS machine.

Next you will want to make sure that both the EMS and the TMG Array Server trust the CA that issued the Server Authentication certificate to the EMS. You can do this by importing the certificate for the CA into the Trusted Root Certificate Authorities branch in the Computer Store on each of those machines.

Another thing that is sometimes overlooked in the scenario is that mirrored accounts are needed on both the EMS and the TMG Array Server. For example, I just used the fabrikam/administrator account on the EMS and the local Administrator account on the TMG Array server. They both have the same password.

Keep in mind that if you have any firewalls that reside between the EMS and the TMG Array Servers you will initially want to allow ALL traffic between them. You can tweak this down later but it can cause you a lot of heartache with communications in TMG.

Create the New Array

On your EMS, open the TMG MMC, highlight the Arrays branch, and then on the far right-hand Tasks pane choose to Create New Array

Give your new array a name.

Type in the DNS name of the array.

Choose the Default Policy.

Click Next at the Array Policy Rule Types.

Complete the New Array Wizard.

TMG will create the new array and you should see that it was a success.

Apply this on the EMS.

Wait for the configuration changes to be saved.

In the TMG MMC on your EMS, there should now be a branch called Arrays. Below it should be the array that you just created.

Joining the New Array

Back on your TMG Array Server go into the TMG MMC and highlight the branch that says Forefront TMG (servername). On the far right-hand pane under the Tasks Tab, click Join Array.

You will see a welcome screen for the Join Array Wizard.

Under the Array Membership Type choose to “Join an array managed by an EMS Server”.

Give it the Fully Qualified Domain Name of your EMS. (Note: you will want to make sure name resolution is working properly on the TMG Array Server before you do this step).

The newly created array should come up as a choice.

Click Finish on the Completing the Join Array Wizard.

You should get a message that you successfully joined the array.

Give it a few minutes but you will probably notice that the configuration is not synching and you will get an error and a red X under the Configuration Status. The error reads “Forefront TMG Management cannot establish a connection with the Forefront TMG Computer.”

The Problem

So why isn’t the TMG Array Server able to communicate with the EMS? It seems like everything was set up correctly. TMG in a workgroup scenario relies on Authentication over SSL encrypted channel (LDAPS). That is the reason we requested the Server Authentication certificate for the EMS Server.

You can verify this by going into the MMC on your EMS, right-clicking on the top level branch of the array that you just created and choosing Properties.

Under the Configuration Storage branch the authentication type near the bottom should be set to “Authentication over SSL encrypted channel”

The problem is that the Server Authentication certificate was never bound to the ISASTGCTRL service.

You can verify this by creating a Certificates Snap-in MMC.

Choose to manage snap-in for a Service Account.

Select the Local Computer.

Select the ISASTGCTRL service and finish.

You should now see that there is not a certificate under the Personal branch.

Keep this open, you will refresh it in a few minutes.

To correct the certificate issue, download the TMG Cert Tool Pack from here.

Install the tool on the EMS but then move the ISACertTool.exe to the same directory where TMG is installed. Open an administrative command prompt and navigate to that directory. Run the command as explained below against the .PFX file you have for your EMS Server Authentication Certificate.

The syntax is listed in the DOC file that comes with it and is:

• ISACertTool /st file_name [/pswd password] [/keepcerts]

Where:

• /st file_name installs the exported certificate on the Configuration Storage server. File_name specifies the path and name of the exported .pfx certificate file.

• /pswd password specifies the password that may be required when installing the server certificate. It is only required if a password was specified during export of the certificate file.

• /keepcerts specifies that existing certificates should not be deleted. By default when you run ISACertTool.exe, all certificates in the ADAM_ISSTGCTRL local store are erased. To specify that existing certificates should not be deleted, specify the /keepcerts parameter.

After running this you should see a message that the Storage certificate installation was successful.

Go back to the Certificate Snap-in for the ISASTGCTRL Service and refresh the Personal Branch. You should now see the Server Authentication Certificate.

Now go back to Monitoring on your EMS and you should see that the TMG Server is successfully synching.

Conclusion

Setting TMG up where the EMS is in a domain and the TMG Array Servers are in a workgroup can be tricky. TMG in a workgroup relies on an SSL Encrypted channel and sometimes getting that to work correctly is not always straight-forward. In this article I have shown you a couple of the common pitfalls and how to correct them.

Author

Keith Abulton – Security Support Escalation Engineer, Microsoft CTS Forefront Security Edge Team

This post is about an issue that I came across, while working on a case recently.

SCENARIO:

The scenario was a simple website publishing through ISA server 2006. While accessing the HTTPS website from a client, we were seeing a Failed Connection attempt in the ISA logs. More specifically, the error message was pointing to Error code 13, “the data is invalid”.  First, we made sure the rule elements were configured correctly.  Next, while reproducing the issue, we collected an ISA Data Packager with the “Web Proxy and Web Publishing” template selected…and also collected a decrypted SSL Network trace from the ISA server.

DATA ANALYSIS:

From the decrypted SSL trace, we could identify that the ISA server returned a response code of 400 (BAD REQUEST). Upon further inspection of the HTTP headers and payload (the request was a POST), we could not find any issue which was obvious. So we looked further into the ISA Tracing file, which is a component level trace collected by the ISA Data Packager.

In that trace, the request seemed to contain a Multi-Line header which is not allowed by ISA server, by default. Hence it is rejecting the request made by the client.  The ISA tracing contained the following error:

ERROR:Multi line header - unsupported

We then revisited the decrypted SSL trace and observed that there was a MultiLine header found.

This trace example contains a multi-line header

In an attempt to reproduce the issue, we also used a web debugging tool to construct a similar HTTP POST request with the same payload and same headers and that worked as expected!  When we checked the network trace from our repro, we could see that the header was not sent as a “Multi-line”.

This trace example does not contain a multi-line header

The following byte codes (highlighted above) can be indicative of a multi-line header:

0d – Carriage Return

0a – New line (Line Feed)

09 – Horizontal tab

More information about the byte codes and their corresponding mappings can be found in RFC2616.

RESOLUTION:

In order to resolve the issue, we opened the registry editor on the problematic server and navigated to HKEY_LOCAL_MACHINE\SOFTWARE\MICROSOFT\RAT\Stingray\Debug\Web Filters

and added a DWORD value APPEND_CONTINUATION_LINES and set the value to 1, and then rebooted the server.

The issue no longer occurred. The client was able to access the site successfully.

The solution to this issue has been addressed in detail in the following KB article:

http://support.microsoft.com/kb/941293

This fix appends the continuation tab, thereby enabling the ISA server to successfully read the headers and process it further.

I hope this post would be helpful in resolving issues that you may face while publishing web applications through ISA/TMG.  But always remember not to make any changes to the registry unless you have isolated the root cause of the issue. Also, as a general best practice, take a backup of the registry before making modifications.

Author:

Karthik Divakaran

Security Support Engineer – MSD Security Team

Reviewers:

Junaid Jan

Security Support Escalation Engineer - MSD Security Team

Richard Barker

Sr. Security Support Escalation Engineer - MSD Security Team

As you may know, there is a known security vulnerability for the authentication method MS-CHAPv2.

The following TechNet article provides some detailed information about it:
Microsoft Security Advisory (2743314)

Unencapsulated MS-CHAP v2 Authentication Could Allow Information Disclosure

http://technet.microsoft.com/en-us/security/advisory/2743314

You may consider moving away from PPTP VPN connections which are configured to use this authentication method therefore.

But, what are the alternatives and how can you integrate these in TMG 2010 which is configured as a VPN Server?

As you can see, only VPN solutions that rely on PPTP in combination with MS-CHAP v2 as the sole
authentication method are vulnerable to this issue. If you need to rely on PPTP connections due to down level clients’ requirements, you may consider changing the authentication method for such PPTP connection to PEAP-MSCHAPv2. I will illustrate later how this can be achieved with TMG 2010.

First of all I would like to mention that you should consider using SSTP or L2TP/IPSec connections instead of PPTP.

Both types are also supported and integrated in TMG 2010.

Why you should deprecate PPTP?

PPTP uses Microsoft Point to Point encryption (MPPE). MPPE uses RC4 stream cipher for encryption. There is no method for authentication of the cipher text stream and therefore the cipher text is vulnerable to a bit-flipping attack.

You could use an authentication method like EAP-TLS, PEAP-TLS or PEAP-MSCHAPv2 for a PPTP connection to address the mentioned vulnerability in MS-CHAPv2. But the encryption method does not change by doing that.

So let’s take a look at L2TP/IPsec and SSTP.
The main advantage of Secure Socket Tunneling Protocol (SSTP) is that it is uses TCP port 443 as its server destination port since it relies on SSL which provides transport layer security. Therefore the SSTP packets can pass firewalls and proxies. Unfortunately there is no built-in SSTP support in down level or several 3^rd party operating systems. That’s why the affected administrators need to decide between PPTP and L2TP/IPsec for compatibility reasons.

L2TP/IPsec is well-known and has been widely used for many years. If you need some in-depth information, please refer to the following article:

http://technet.microsoft.com/de-de/library/cc771298(v=ws.10).aspx

Because of that fact that there is no need to reinvent the wheel, I would like to point to Deb Shinder who did a pretty good job writing a step-by-step guide on how to implement L2TP/IPsec in TMG 2010.

http://www.isaserver.org/tutorials/Checking-Out-TMG-2010-Virtual-Private-Network-Server-Part3.html

Though its quality, Microsoft does not adhere for third party information.

Before we look at the configuration of PEAP-MSCHAPv2, let me clarify that PEAP-MSCHAPv2 is NOT affected by the mentioned vulnerability, because the MSCHAPv2 information is only transmitted within a secure channel in PEAP phase 2.
The following article provides further information about the authentication method PEAP-MSCHAPv2 in general:

http://technet.microsoft.com/en-us/library/cc779326(v=WS.10).aspx
One more word before you read ahead. PPTP PEAP-MSCHAPv2 requires a certificate with Server Authentication EKU on a NPS Server from either your organizations PKI or a public CA.

Now let’s focus on how to implement PEAP-MSCHAPv2 in TMG. You may expect that TMG supports PEAP-MSCHAPv2 as valid authentication method, since it is some sort of EAP variant with improved security and because EAP can be enabled in TMG as authentication method. Unfortunately this is not 100% correct. If you choose EAP it in the Remote Access Policy it will cause the VPN connection to fail.

A Windows 7 client will report the following error:

Why does this happen?
The reason is that TMG configures the Routing and Remote Access Service (RRAS) and the Network Policy Service (NPS) which are included in the Windows Server operating system.

Doing this, TMG adds the following authentication method as a valid type to its local NPS configuration:
“Microsoft: Smart Card or other certificate” which is EAP-TLS, but neither PEAP-MSCHAPv2 or PEAP-TLS.

If you change the NPS configuration on the TMG Server, it will only provide a temporary workaround. The reason is that TMG will overwrite the NPS configuration once its service is restarted.

How you can solve that:

You will need to configure TMG to use Radius Authentication. This will result in outgoing Radius packets to a remote Radius/NPS Server. This additional NPS Server can be configured to enable PEAP support.

As next step you will need to define the remote RADIUS Server(s)

This is it from TMG perspective. After you have applied the configuration, you will need to configure the remote NPS Server.

After you have added the required NPS role on your remote server, open the NPS MMC.

First of all you will need to define and enable a RADIUS client there. This will ensure that incoming RADIUS packets from this source will not be dropped.

Next you will need to create a ‘Connection Request Policy’ and a ‘Network Policy’ in the remote NPS Server.

The following article explains the differences between ‘Connection Request Policies’ and ‘Network Policies’.

http://technet.microsoft.com/en-us/library/cc772279(v=WS.10).aspx

When installing NPS, you will have some policies already. Either you can edit them or create some new ones. We will create new ones for this example. Let’s start with a new ‘Connection Request Policy’ by using the wizard.

Then you will have to define conditions when this policy applies. You have lots of possibilities to define granular rules which match your needs. These possibilities are illustrated in the linked article above. To make things as easy as possible for this guide, we only use a Day-and-Time condition. Basically the condition always applies.

Leave the default settings on the next screen. You could forward the requests once more (Radius Proxy feature), but that does not make sense for this example.

If you are using Network Access Protection (NAP) for your VPN clients, you will need to use the setting “Override network policy authentication settings”. If you don’t use NAP, leave the default settings which I do for this straight-forward example.

Leave the settings on the next page to its defaults and click ‘Next’. After you have checked the summary, click on ‘Finish’. That’s it for the ‘Connection request policy’.

Let’s move on to the network policy.

Right click on ‘Network Policies’ and choose ‘New’.

In the next screen define a suitable policy condition which defines, when the network policies is about to be applied for a connection request. Again, you have plenty of options to use granular conditions. This does not only make sense, but may also be required due to your infrastructure needs. For example you may want to use different authentication methods, you may need to create different policies for VPN, 802.1x, etc.

To keep things simple, we use the Day and Time restriction once more. Again, using it in this way, it will always apply this rule.

Skip the next screen by clicking on ‘Next’. This will bring you to the configuration of the valid EAP types.

You will need PEAP-MSCHAPv2 for this scenario. Uncheck the ones which are not required and add PEAP-MSCHAPv2 by clicking on ‘Add’.

Click on ‘Ok’.

You screen should look like this:

Highlight the PEAP entry in the list and click ‘Edit’:

Make sure that you have chosen a valid and trusted certificate. Besides add EAP-MSCHAP v2 as inner method.

Click ‘Ok’, click ‘Next’ three times and then ‘Finish’ completing the configuration.

That’s it. I hope this guide helps you to understand if you need to change your VPN type and if needed, how to do it.

I am happy to answer related questions and read your feedback in the comments below.

Author:

Frank Hennemann

Microsoft CSS Forefront Security Edge Team

Technical Review:

Markus Sarcletti

Microsoft CSS Windows Networking

Sergio Medina here, and today I want to talk about a question we receive every now and then and explain what the solution is just in case some of you run into a similar issue. You could hit this issue when publishing an internal webserver that accepts only the username as a valid format. For example, when your web server is configured this way, if you try to log in directly to it using ‘DOMAIN\user’ as the format for the username you receive an error, however if you only include the ‘user’ part of the name in the user field, and provide a valid password of course, you log in successfully.

So let’s say you have an internal web server configured this way and you just configured your new web publishing rule in a Forefront Threat Management Gateway 2010 (TMG 2010) server/array. You used Form Based Authentication (FBA) as the authentication method in the web listener and HTTP Basic delegation in the web publishing rule.

Unfortunately, when you try to test your new publishing rule you find that it is not working as expected. If you analyze network traces between the TMG 2010 server and the webserver you discover that TMG 2010 is adding the NetBIOS domain name as part of the user name value:

Now we see the cause of the problem:

· The user introduces his/her username in the TMG form, as ‘user’
· TMG 2010 adds the NetBIOS domain name, sending the authentication in the format ‘DOMAIN\user’
· The webserver only accepts ‘user’ as a valid format so the request fails

So what could we do now?

a) Change webserver behavior, accepting also ‘DOMAIN\user’.

or

b) Change TMG behavior, sending only the username introduced by the user.

Solution a) is out of the scope of this blog article, so let’s see how to achieve solution b).

The trick here is to change the value of StripDomainFromCredentials, one of the internal properties of the web publishing rule. This property is not accessible using the Forefront TMG Management console so we need to change this is by using COM objects.

NOTE TheIFPCWebPublishingProperties2::StripDomainFromCredentials Property is documented here:http://msdn.microsoft.com/en-us/library/ff827099(v=vs.85).aspx

So with a simple script we could change this property for any web publishing rule. Here are the steps:

1. Copy the following lines into a new file named StripDomain.vbs on the TMG 2010 server:

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
' Microsoft Customer Technical Support
' Script for enabling StripDomainFromCredentials on a rule in TMG 2010  
' THIS CODE IS MADE AVAILABLE AS IS, WITHOUT WARRANTY OF ANY KIND. THE ENTIRE
' RISK OF THE USE OR THE RESULTS FROM THE USE OF THIS CODE REMAINS WITH THE
' USER. USE AND REDISTRIBUTION OF THIS CODE, WITH OR WITHOUT MODIFICATION, IS
' HEREBY PERMITTED.
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
Option Explicit  
Dim WebPublishingRuleName, newStripDomainFromCredentials  
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''  
' SET YOUR VALUES HERE  
' Rule name to change  
WebPublishingRuleName = "Rule name"  
' Set here custom values  
newStripDomainFromCredentials = True  
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''  
' Begin  
Dim Root, Array, Rules, Rule, intCompare  
Set Root = CreateObject("FPC.Root")  
Set Array = Root.GetContainingArray  
Set Rules = Array.ArrayPolicy.PolicyRules  
''''''''''''''''''''''''''''''''''  
' Look for the WebListener  
For Each Rule in Rules  
Wscript.Echo " Comparing Rule name |" & WebPublishingRuleName & "| with |" & Rule.Name & "|"  
intCompare = StrComp(WebPublishingRuleName, Rule.Name, vbTextCompare)  
If intCompare = 0 then  
Exit For  
End If  
Next  
Wscript.Echo  
Wscript.Echo "Found Rule with description: |" & Rule.Description & "|"  
''''''''''''''''''''''''''''''''''  
' Show values  
Wscript.Echo  
Wscript.Echo "***** CURRENT VALUES: "  
Wscript.Echo "** StripDomainFromCredentials = |" & Rule.WebPublishingProperties.StripDomainFromCredentials & "|"  
Wscript.Echo "***** NEW VALUES: "  
Wscript.Echo "** StripDomainFromCredentials = |" & newStripDomainFromCredentials & "|"  
''''''''''''''''''''''''''''''''''  
' Warning and ask to continue  
Dim strMessage  
WScript.Echo ' newline  
Wscript.Echo "Please check if the previous information is correct and you want to apply the changes"  
strMessage = "Press any key to continue or Ctrl+C to cancel"  
WScript.Echo ' newline  
WScript.StdOut.Write strMessage  
Do While Not WScript.StdIn.AtEndOfLine  
Input = WScript.StdIn.Read(1)  
Loop  
''''''''''''''''''''''''''''''''''  
' Set new values  
Rule.WebPublishingProperties.StripDomainFromCredentials = newStripDomainFromCredentials  
Wscript.Echo "***** CURRENT VALUES: "  
Wscript.Echo "** StripDomainFromCredentials = |" & Rule.WebPublishingProperties.StripDomainFromCredentials & "|"  
Rule.Save  
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

2. Modify the script, changing the value WebPublishingRuleName to the name of the rule you want to modify.

3. Open a command prompt with elevated privileges (Run as Administrator)

4. Run the script using: cscript /nologo StripDomain.vbs

5. Follow the steps in the script.

6. In an Enterprise environment, wait until the configuration changes are applied (Monitoring > Configuration > all servers in the array with Synced Status)

After this, you should see the TMG 2010 server sending only the username, thus resolving our issue:

If you need to undo the change, just edit the script, change the following variable to False and execute it again:

newStripDomainFromCredentials = False

One last thing to mention is that this default behavior has changed from ISA 2006 to TMG 2010 so you might have published your webserver successfully using ISA 2006 but found this new behavior when using TMG 2010.

Hope this helps,

Author: Sergio Medina | Senior Support Engineer | Microsoft CSS Forefront Edge Team

Technical Reviewer: Ashish Kapila | Support Escalation Engineer | Microsoft CSS Forefront Edge Team

Get the latest System Center news onFacebookandTwitter:

System Center All Up: http://blogs.technet.com/b/systemcenter/
System Center – Configuration Manager Support Team blog: http://blogs.technet.com/configurationmgr/
System Center – Data Protection Manager Team blog: http://blogs.technet.com/dpm/
System Center – Orchestrator Support Team blog: http://blogs.technet.com/b/orchestrator/
System Center – Operations Manager Team blog: http://blogs.technet.com/momteam/
System Center – Service Manager Team blog: http://blogs.technet.com/b/servicemanager
System Center – Virtual Machine Manager Team blog: http://blogs.technet.com/scvmm

Windows Intune: http://blogs.technet.com/b/windowsintune/
WSUS Support Team blog: http://blogs.technet.com/sus/
The AD RMS blog: http://blogs.technet.com/b/rmssupp/

The Forefront Endpoint Protection blog : http://blogs.technet.com/b/clientsecurity/
The Forefront Identity Manager blog : http://blogs.msdn.com/b/ms-identity-support/
The Forefront TMG blog: http://blogs.technet.com/b/isablog/
The Forefront UAG blog: http://blogs.technet.com/b/edgeaccessblog/

Hello Everyone! We’ve seen a few cases lately dealing with TMG servers sourcing outgoing packets with secondary IP addresses that have been added to the NICs. This could cause issues in communications between nodes or possibly other issues. One such example that I have seen come across is where a customer had a TMG server being utilized as an internal firewall behind a 3^rd party Edge firewall. Clients were utilizing the TMG server as their proxy server. When the http requests left the external interface of the TMG server the packets were sourced with a secondary IP address of the External NIC on the TMG instead of the primary address of that NIC. When the Edge firewall received the packets it dropped them because its rules were configured to only allow packets out when sourced with the primary IP address of the TMG’s external interface. This of course broke internet connectivity for all internal clients.

The question at hand is… “Why is the TMG server sourcing packets with a secondary address instead of the primary address of the NIC?”

The answer to that question deals with the differentiation between the Network Stack in Server 2008 and above and Server 2003 and below. Server 2003\XP and below were based off the Weak Host Model. Basically, in a Weak Host Model the primary address of the adapter with a route that most closely matches the target IP address is used as the Source IP Address.

In server 2008\Vista and above we re-architected the Network stack. It is based on the Strong Host Model. In the Strong Host Model the concept of a Primary IP Address doesn’t exist. To determine the IP address that is utilized it looks at the routing table to decide the proper NIC to utilize, then uses the process defined in RFC 3484 to choose the source IP for outbound packets. Here is a basic breakdown of how the windows implementation of RFC 3484 chooses an IP address:

Windows Source IP V4 address selection:

- Rule 1 Prefer same address (applies)

- Rule 2 Prefer appropriate scope (applies)

- Rule 3 Avoid deprecated addresses (applies)

- Rule 4 - Prefer home addresses - does not apply to IP v4

- Rule 5 Prefer outgoing Interfaces (applies)

- Rule 6 Prefer matching label - does not apply to IP v4

- Rule 7 Prefer public addresses - does not apply to IP v4

- Rule 8a: Use longest matching prefix with the next hop IP address. (not in RFC!)

"If CommonPrefixLen(SA, D) > CommonPrefixLen(SB, D), then prefer SA. Similarly, if CommonPrefixLen(SB, D) > CommonPrefixLen(SA, D), then prefer SB."

*This says that the IP with the most high order bits that match the destination of the next hop will be used.

Note: Rule 8 - Use longest matching Prefix is similar to rule 8a except the match is with the destination IP address rather than the next hop IP address.

For example, use the following addresses as an example of choosing the longest matching prefix:

TMG Servers External IP Addresses:

192.168.1.14/24
&
192.168.1.68/24

Default Gateway:

192.168.1.127

Convert these addresses into binary:

192.168.1.14   = 11000000 10101000 00000001 00001110
192.168.1.68   = 11000000 10101000 00000001 01000100
192.168.1.127 = 11000000 10101000 00000001 01111111

The 192.168.1.68 address has more matching high order bits with the gateway address 192.168.1.127. This would cause the server to utilize what was originally defined as the “secondary” as the Source IP address of outgoing packets.

*For more information on RFC 3484 please refer to the following link: http://www.ietf.org/rfc/rfc3484.txt . Please note that IPv6 is referenced in RFC. Windows utilize the same process for IPv4 sourcing.

You can also review the following TechNet article for supported document details on the above information:

The functionality for source IP address selection in Windows Server 2008 and in Windows Vista differs from the corresponding functionality in earlier versions of Windows
http://support.microsoft.com/kb/969029

So now we know why your TMG server may be sourcing your packets with what you call your “Secondary IP Address”. It isn’t TMG at all. It is the default behavior of the server itself. Your server version is Server 2008 or above. The question is…

“Can I configure my Server 2008 or above in a way that it will only utilize the first IP address as a Source address?”

The answer to that is YES! There is actually a Netsh command that can be utilized to add IP addresses.  In that command you use the “SkipAsSource” flag and it will no longer use the IP address you are adding as a Source IP Address. This means that you will have to temporarily remove the IP Address you are having the issues with then re-add them utilizing the Netsh command (This means you will have to have a maintenance window!). Here are examples of the command lines you will use:

Server 2008:
Netsh int ipv4 add address <Interface Name> <ip address> <subnet mask> skipassource=true

Server 2008 R2:

Netsh int ipv4 add address <Interface Name> <ip address> skipassource=true

* For details and prerequisites to utilize these commands please refer to the following articles:

All IP addresses are registered on the DNS servers when the IP addresses are assigned to one network adapter on a computer that is running Windows Server 2008 SP2 or Windows Vista SP2
http://support.microsoft.com/default.aspx?scid=kb;EN-US;975808

IP addresses are still registered on the DNS servers even if the IP addresses are not used for outgoing traffic on a computer that is running Windows 7 or Windows Server 2008 R2
http://support.microsoft.com/default.aspx?scid=kb;en-US;2386184

Keep in mind that I gave only one specific example where this may be causing an issue.  There may be other problems you run into where the Netsh entry I listed may help you out.  No telling… it may not even be on your TMG servers.  Maybe you see the issue on your UAG servers, or any other server for that fact. 

I hope the information provided helps out!

Author

Brett Crane - Sr Security Support Escalation Engineer, Microsoft CTS Forefront Security Edge Team

Reviewer

Richard Barker - Sr Security Support Escalation Engineer, Microsoft CTS Forefront Security Edge Team

This post is, once again, about an issue I worked on few days back.  Before I start discussing the issue, and how I resolved it, I would like to outline the objective of this post.

The objective of this post is to make TMG administrators aware of issues like this; and what can be done to resolve them. Discovering the root cause of this issue required a User Mode Dump analysis.

Performing a User Mode Dump analysis requires “Symbol” files (which are private). My goal is not to provide specific instruction on User Mode Dump analysis, but instead to show what kind of information can be gathered, and how it can be used, to help troubleshoot boot-time “service issues” on a TMG server.

For those that are not familiar with dump analysis terms like process, threads and its stack, I will elaborate further as I explain the steps.

Issue:

TMG server admin was rebooting the server and at the time of reboot TMG services were hanging and were not starting. A similar issue was reported pre TMG sp2 but it was fixed post sp2. In this scenario TMG was updated to latest build i.e. TMG sp2 RU2.

Troubleshooting:

Some background: It should be noted that quite a bit of troubleshooting had taken place prior to my involvement in the case. This includes the steps in the following Knowledge Base article:

Forefront Threat Management Gateway 2010 services do not start as expected when the FTMG 2010 servers are in a workgroup array

During startup, the following System Event was logged…

_____________________________________________________________________________________________

Log Name:      System
Source:        Service Control Manager
Date:          09/11/2012 17:42:30
Event ID:      7022
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      server1
Description:
The Microsoft Forefront TMG Firewall service hung on starting.

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
    <EventID Qualifiers="49152">7022</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2012-11-09T17:42:30.378163900Z" />
    <EventRecordID>344470</EventRecordID>
    <Correlation />
    <Execution ProcessID="716" ThreadID="720" />
    <Channel>System</Channel>
    <Computer>server1</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="param1">Microsoft Forefront TMG Firewall</Data>
  </EventData>
</Event>

_____________________________________________________________________________________________

Data collection:

During the course of troubleshooting we collected a User Mode Dump while reproducing the issue.

User mode dumps collection reference: http://msdn.microsoft.com/en-us/library/ff420662.aspx

Data analysis:

Note: The approach taken in this post is very similar to guidelines given in the following link about debugging a deadlock as we were in a scenario similar to a deadlock: http://msdn.microsoft.com/en-us/library/windows/hardware/ff540592(v=vs.85).aspx

In the dump, I found following critical section was locked :

Note: For more information about critical section and locked critical section, please refer to: http://msdn.microsoft.com/en-us/library/windows/hardware/ff541979(v=vs.85).aspx

Then I located the owning thread of this locked critical section. In following snapshot we can see the stack of this thread. The stack is read from bottom to top. From this call stack it appears that wspsrv (firewall service) is trying to load a filter called XSISAPI. It appears TMG has deferred its filters’ startup until this filter (i.e. XSISAPI)is loaded.

I then checked the module for this filter (i.e. XSISAPI) and found that it’s a filter called “Afaria” from Sybase.

Solution:

We configured the XSISAPI filter service to delayed start. After this change, the TMG services started normally after reboot.

Author:

Suraj Singh:

Security Support Escalation Engineer - MSD Security Team

Reviewer:

Richard Barker

Sr. Security Support Escalation Engineer - MSD Security Team