Messaging subsystem broken?

Continua CI Bugs
1

Hi guys,

since the update to 1.9.0.188 the messaging subsystem in Continua CI is broken. We don’t get any private messages or emails. The event log shows a lot of error messages like this:

An unhandled exception occurred while NotificationQueueController was consuming a message of type BuildStateChangedMessage: NHibernate.Exceptions.GenericADOException: could not execute query 
[ select usernotifi0_.Id as id1_24_, usernotifi0_.UserId as userid2_24_, usernotifi0_.PlugInName as pluginname3_24_, usernotifi0_.PlugInValue as pluginvalue4_24_ from core_usernotificationpreferences usernotifi0_ where usernotifi0_.UserId=@p0 ] 
Name:p1 - Value:a608ccc4-8a9a-4fd2-a825-a8f201049236 
[SQL: select usernotifi0_.Id as id1_24_, usernotifi0_.UserId as userid2_24_, usernotifi0_.PlugInName as pluginname3_24_, usernotifi0_.PlugInValue as pluginvalue4_24_ from core_usernotificationpreferences usernotifi0_ where usernotifi0_.UserId=@p0] ---> System.InvalidOperationException: ExecuteReader requires an open and available Connection. The connection's current state is closed. 
at System.Data.SqlClient.SqlCommand.ValidateCommand(String method, Boolean async) 
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) 
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) 
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method) 
at NHibernate.AdoNet.AbstractBatcher.ExecuteReader(DbCommand cmd) 
at NHibernate.Loader.Loader.GetResultSet(DbCommand st, QueryParameters queryParameters, ISessionImplementor session, IResultTransformer forcedResultTransformer) 
at NHibernate.Loader.Loader.DoQuery(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies, IResultTransformer forcedResultTransformer) 
at NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollections(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies, IResultTransformer forcedResultTransformer) 
at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters, IResultTransformer forcedResultTransformer) 
--- End of inner exception stack trace --- 
at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters, IResultTransformer forcedResultTransformer) 
at NHibernate.Loader.Loader.ListUsingQueryCache(ISessionImplementor session, QueryParameters queryParameters, ISet`1 querySpaces) 
at NHibernate.Hql.Ast.ANTLR.QueryTranslatorImpl.List(ISessionImplementor session, QueryParameters queryParameters) 
at NHibernate.Engine.Query.HQLQueryPlan.PerformList(QueryParameters queryParameters, ISessionImplementor session, IList results) 
at NHibernate.Impl.SessionImpl.List(IQueryExpression queryExpression, QueryParameters queryParameters, IList results, Object filterConnection) 
at NHibernate.Impl.SessionImpl.List(IQueryExpression queryExpression, QueryParameters queryParameters, IList results) 
at NHibernate.Impl.AbstractSessionImpl.List(IQueryExpression queryExpression, QueryParameters parameters) 
at NHibernate.Impl.AbstractQueryImpl2.List() 
at NHibernate.Linq.DefaultQueryProvider.ExecuteQuery(NhLinqExpression nhLinqExpression, IQuery query, NhLinqExpression nhQuery) 
at NHibernate.Linq.DefaultQueryProvider.Execute(Expression expression) 
at NHibernate.Linq.DefaultQueryProvider.Execute[TResult](Expression expression) 
at Remotion.Linq.QueryableBase`1.GetEnumerator() 
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) 
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) 
at Continua.Membership.UserManager.InternalGetUserNotificationPreferencePlugIns(Guid userId) 
at Continua.Membership.UserManager.<GetNotificationUserPreferencesPlugIn>b__25_0(Guid id) 
at Continua.Membership.UserPreferencesPlugInCache.GetPlugIn(Guid userId, String plugInName, Func`2 getUserPreferencesFunc) 
at Continua.Membership.UserManager.GetNotificationUserPreferencesPlugIn(String plugInName, Nullable`1 userId, User user) 
at Continua.Notification.NotificationQueueController.AreUserPreferencesValid(String plugInName, User user, String messageDetails) 
at Continua.Notification.NotificationQueueController.PublishMessage(Subscription subscription, INotificationModel notificationModel, IList`1 matchedSubscriptions, ISession session) 
at Continua.Notification.NotificationQueueController.Consume(IMessage message) 
at Continua.Shared.Messaging.MessageBroker.SendMessage[T](Object args)

Do you guys have any idea to fix this? The changelog for the latest version does not say anything about fixes in the messaging subsystem.

With kind regards

Kay Zumbusch

2

Hi Kay,

We’ll investigate this today. Can you let us know which version of SQL server you are using?

3

Hi Kay,

We have not been able to reproduce this error in our testing today.

Can you check whether there is another different error in the event log before this one, as this error states that the database connection is closed - and this is perhaps because of a prior exception?

Otherwise, or additionally in case the other error does not lead us to a diagnosis, can you enable debug logging? Then run a build or make a change, to publish a message and cause the error. Then send the debug log via a direct message to me.

4

Hi Dave,

to get the debug log I restarted the server. After the restart the messaging subsystem works fine. As we already had debug logging enabled, I’ll check the debug log for exceptions and will get back to you if I find exceptions that are not logged by the messaging subsystem.

With kind regards

Kay Zumbusch