Detailed Message:
TF271001: An error occurred while attempting to send an e-mail notification to the following address: email id. Further e-mail notification errors that occur within the next five minutes might not be logged. Verify that the e-mail notification settings are correct in the Team Foundation Administration Console.
Exception Message: Service not available, closing transmission channel. The server response was: 4.4.1 Connection timed out (type SmtpException)
Error :
TFS Email Notifications Delayed or Not Sent
Explanation:
I started experiencing problems with TFS alert notification mails. TFS Email Notifications were Delayed or Not received. The TFS alert notification was working fine for the past 2 years.
Solution:
Following are the investigation done to fix the notification problem
1. Verified the SMTP server and found out no problems with it. (Anonymous sender was enabled in Exchange for tfs@domain.com)
2. Verified whether "Visual Studio Team Foundation Background Job Agent" service is running in TFS App tier. Service was running fine. (Restarted the service too)
3. I ran the follwoing query to find tfs job failures in the tbl_JobHistory table in the Tfs_Configuration database
SELECT TOP 10 DATEADD(hour, -7, StartTime) as StartTimePDT, Result, ResultMessage
FROM [Tfs_Configuration].[dbo].[tbl_JobHistory] WHERE JobId = 'A4804DCF-4BB6-4109-B61C-E59C2E8A9FF7' AND Result <> 0
ORDER BY StartTime DESC
Note: The JobId GUID for the “Team Foundation Server Event Processing” job with the namespace: “Microsoft.TeamFoundation.JobService.Extensions.Core.NotificationJobExtension” is 'A4804DCF-4BB6-4109-B61C-E59C2E8A9FF7'
Query result:
"There were errors or warnings during notification delivery. 35573/35598 emails delivered. 0/0 soap notifications delivered. 25 errors. 0 warnings. Set /Service/Integration/Settings/NotificationJobLogLevel in the TF registry to 2 to see errors and warnings"
4. Queried [dbo].[tbl_EventNotification] table in the [Tfs_DefaultCollection] database to see for failures.
SELECT * FROM [Tfs_DefaultCollection].[dbo].[tbl_EventNotification]
Found many records with failed status.
5. Deleted all records with failed status
Delete FROM [Tfs_DefaultCollection].[dbo].[tbl_EventNotification] where [status] = 'failed'
Note : I dint see any harm in deleting all the records from FROM [Tfs_DefaultCollection].[dbo].[tbl_EventNotification]
Delete FROM [Tfs_DefaultCollection].[dbo].[tbl_EventNotification]
6. Queried [dbo].[tbl_EventNotification] table in the [Tfs_DefaultCollection] database to see the notification event backlogs count.
SELECT * FROM [Tfs_DefaultCollection].[dbo].[tbl_ClientEvent]
Found around 1000 pending events that need to be processed.
7. Deleted all pending events backlogs
Delete FROM [Tfs_DefaultCollection].[dbo].[tbl_ClientEvent]
Verified the fix by trigging the new alerts, All the alerts got processed and also received the notification mails.
link to turn on verbose log
link to Decrease or Increase notification delay settings in TFS
Subscribe