TF271001: The server response was: 4.4.1 Connection timed out type SmtpException. TFS Email Notifications Delayed or Not Sent

by XDK 17. July 2013 19:53

Detailed Message:

TF271001: An error occurred while attempting to send an e-mail notification to the following address: email id. Further e-mail notification errors that occur within the next five minutes might not be logged. Verify that the e-mail notification settings are correct in the Team Foundation Administration Console.
Exception Message: Service not available, closing transmission channel. The server response was: 4.4.1 Connection timed out (type SmtpException)

Error :

TFS Email Notifications Delayed or Not Sent

Explanation:

I started experiencing problems with TFS alert notification mails. TFS Email Notifications were Delayed or Not received. The TFS alert notification was working fine for the past 2 years.

Solution:

Following are the investigation done to fix the notification problem

1. Verified the SMTP server and found out no problems with it. (Anonymous sender was enabled in Exchange for tfs@domain.com)
2. Verified whether "Visual Studio Team Foundation Background Job Agent" service is running in TFS App tier. Service was running fine. (Restarted the service too)


3. I ran the follwoing query to find tfs job failures in the tbl_JobHistory table in the Tfs_Configuration database

SELECT TOP 10 DATEADD(hour, -7, StartTime) as StartTimePDT, Result, ResultMessage
FROM [Tfs_Configuration].[dbo].[tbl_JobHistory] WHERE JobId = 'A4804DCF-4BB6-4109-B61C-E59C2E8A9FF7' AND Result <> 0
ORDER BY StartTime DESC

Note: The JobId GUID for the “Team Foundation Server Event Processing” job with the namespace: “Microsoft.TeamFoundation.JobService.Extensions.Core.NotificationJobExtension” is 'A4804DCF-4BB6-4109-B61C-E59C2E8A9FF7'

Query result:

"There were errors or warnings during notification delivery.    35573/35598 emails delivered.  0/0 soap notifications delivered.   25 errors.  0 warnings.    Set /Service/Integration/Settings/NotificationJobLogLevel in the TF registry to 2 to see errors and warnings"   

4. Queried [dbo].[tbl_EventNotification] table in the [Tfs_DefaultCollection] database to see for failures.

SELECT * FROM [Tfs_DefaultCollection].[dbo].[tbl_EventNotification]

Found many records with failed status.

5. Deleted all records with failed status

Delete FROM [Tfs_DefaultCollection].[dbo].[tbl_EventNotification] where [status] = 'failed'

Note : I dint see any harm in deleting all the records from FROM [Tfs_DefaultCollection].[dbo].[tbl_EventNotification]

Delete FROM [Tfs_DefaultCollection].[dbo].[tbl_EventNotification]

6. Queried [dbo].[tbl_EventNotification] table in the [Tfs_DefaultCollection] database to see the notification event backlogs count.

SELECT * FROM [Tfs_DefaultCollection].[dbo].[tbl_ClientEvent]

Found around 1000 pending events that need to be processed.

7. Deleted all pending events backlogs

Delete FROM [Tfs_DefaultCollection].[dbo].[tbl_ClientEvent] 

Verified the fix by trigging the new alerts, All the alerts got processed and also received the notification mails.

link to turn on verbose log

link to Decrease or Increase notification delay settings in TFS

 

Tags: ,

Azure DevOps on-premises | TFS 2010 | TFS 2012

Decrease or Increase notification delay settings in TFS

by XDK 16. July 2013 21:39

Explanation:

By default notification delay is set to 120 seconds. The notifications are processed in batchs for every 2 minutes. The notification delay can be set in the TF registry
Note: The TFS registry mentioned is not the Windows Registry on the TFS server.

Solution:

The registry value can be changed using the PowerShell script or using the TfsReg tool

Following is the command to set the notification delay to 1 minute using the tool:

tfsreg.exe /server:http://tfs:8080/tfs /path:/Service/Integration/Settings/NotificationJobDelay /value:60

Note: Restart the TFS IIS app pool for the change to take effect.

Tags: ,

Azure DevOps on-premises | TFS 2010 | TFS 2012

mt.exe : general error c101008d: Failed to write the updated manifest to the resource of file "XXXXXXX.exe". Access is denied.

by XDK 6. July 2013 20:11

Exception:

mt.exe : general error c101008d: Failed to write the updated manifest to the resource of file "XXXXXX.exe". Access is denied.

Explanation:

The error may occur in some situations due to a race condition between the compiler process and the anti-virus file-system auto-protect.

Workaround:

 Disable deferred file scanning by adding a registry key.

 

1. Click Start > Run 

2. Type in: regedit and click OK

3. Navigate to: 

"HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Symantec Endpoint Protection\AV\Storages\Filesystem\RealTimeScan" (32 bit Operating System) or

“HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Symantec\Symantec Endpoint Protection\AV\Storages\Filesystem\RealTimeScan” (64 bit Operating System)

4. Click Edit > New > DWORD Value

5. Name the new value: DeferredScanning

6. Open DeferredScanning

7. In the Value data field add “0” (1 = on, 0 = off)

8. Click OK

9. Close the Registry Editor window.

Reference 

Tags:

Team Build

a 64 bit debugging operation is taking longer than expected

by XDK 5. July 2013 09:06

 

 

Solution:

I uninstalled a software named "Sendori".  The error got disappeared.

Discussion about Sendori

http://answers.microsoft.com/en-us/protect/forum/protect_other-protect_scanning

Tags:

Visual Studio 2012

TFS event 7005 and 7006 warnings

by XDK 4. July 2013 10:47

Explanation:

The default threshold is set to 30 seconds, The warning occurs when the threshold exceeds.

Log Name:      Application
Source:        TFS Services
Date:          1/27/2013 8:53:37 AM
Event ID:      7006
Task Category: None
Level:         Warning
Detailed Message: There are no active requests for service host DefaultCollection that exceed the warning threshold of 30.
 
Log Name:      Application
Source:        TFS Services
Date:          1/27/2013 8:52:22 AM
Event ID:      7005
Task Category: None
Level:         Warning
Detailed Message: A request for service host DefaultCollection has been executing for 37 seconds, exceeding the warning threshold of 30.

Solution:

Increase the threshold limit, if you experience performance issues due to these warnings. execute the following SQL statements

use [<Collection DB name>]

DECLARE @registryUpdates typ_KeyValuePairStringTableNullable
INSERT @registryUpdates ([Key], [Value])
SELECT '#\Configuration\ServiceHostMonitor\TotalExecutionElapsedThreshold\', '300'
 
EXEC prc_UpdateRegistry 1, 'Test', @registryUpdates, 1

Tags: ,

Azure DevOps on-premises | TFS 2010 | TFS 2012

About the author

My name is Xavier Dilip Kumar Jayaraj having 16+ years of IT experience which includes solid experience and depth Knowledge in Application Life Cycle Management, Configuration Management, Implementation and Support using TFS on-premises and Azure DevOps. I have invested in gaining DevOps knowledege to expertise with Cloud Computing providers namely Microsoft Azure and Amazon Web Services in recent years. I am very positive to learn and adapt emerging technologies to client’s environment.

Microsoft Certified: Azure Administrator Associate

Microsoft Certified: Azure DevOps Engineer Expert

DevOps Engineer Certificate Program | Transcript 

OTP-AWSD5: AWS IoT: Developing and Deploying an Internet of Things

[PCEP-30-01] PCEP – Certified Entry-Level Python Programmer

Quotes I Like

"Failure will never overtake me if my determination to succeed is strong enough."  - Dr. APJ. Abdul Kalam

"Always be yourself, express yourself, have faith in yourself, do not go out and look for a successful personality and duplicate it." - Bruce Lee

"Technology is just a tool. In terms of getting the kids working together and motivating them, the teacher is the most important." - Bill Gates

"Innovation distinguishes between a leader and a follower." - Steve Jobs

CategoryList

Disclaimer

The information provided here is based on my expreriences, troubleshooting and online/offline findings. It can be used as is on your own risk without any warranties and I impose no rights.