Seq container instance - seq container terminating and restarting

[rodneytamblyn]

I have seq running in a Azure container instances. Tonight the seq container is terminating and restarting (resulting in logging down).

What is the best way to investigate and resolve these issues?

Is there a trouble shooting guide to investigating and restoring service. I would be OK with starting from scratch if necessary, the main thing is to get service restored.

[nblumhardt]

Hi @rodneytamblyn, thanks for you message.

Does az container logs produce any useful output?

There's also a Logs directory in the attached /data mount, which may contain some relevant info, if the container is able to run for long enough to record some.

[rodneytamblyn]

I think the relevant error is as follows. As I needed to restore service I resorted to resetting seq to default (as I have exported/backed up critical log events using JSON exporter).

{"@t":"2024-05-22T13:29:33.1577879Z","@mt":"Failed to start server","@l":"Fatal","@x":"Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Seq.Cluster.IClusterNode -> λ:Seq.Server.StorageSubsystem.\n ---> Flare.Ffi.Result.FlareException: Flare native storage failed (InternalError), data file `\"/data/Documents/metastore.collection.705e5e4a86e54c8d8ca09b2bfffcb116.docc\"` is missing.\n   at Flare.Ffi.Result.FlareResult.EnsureSuccess()\n   at Flare.Ffi.Result.FlareResult.HandleNonSuccessWith(NonSuccessResultHandling handling)\n   at Flare.Ffi.Bindings.StoreOpen(IntPtr path, UIntPtr pathLen, IntPtr limits, StoreHandle& store, NonSuccessResultHandling nonSuccessResultHandling)\n   at Flare.Store.OpenStandalone(String path, StoreLimits limits)\n   at Quince.Storage.Native.NativeDocumentStore..ctor(String path)\n   at Seq.Server.StorageSubsystem.ConstructInnerMetastore(String documentsPath, SeqConfig config)\n   at Seq.Server.StorageSubsystem.Initialize(IComponentContext ctx, SeqConfig config, StorageLayout layout, PerformanceCounters sharedCounters)\n   at Seq.Server.SeqServerModule.<Load>b__9_10(IComponentContext c)\n   at Autofac.RegistrationExtensions.<>c__DisplayClass41_0`1.<Register>b__0(IComponentContext c, IEnumerable`1 p)\n   at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)\n   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)\n   at Autofac.Core.Activators.Delegate.DelegateActivator.<ConfigurePipeline>b__2_0(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)\n   at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)\n   at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   --- End of inner exception stack trace ---\n   at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)\n   at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext context)\n   at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)\n   at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)\n   at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)\n   at Autofac.Extensions.DependencyInjection.KeyedServiceMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)\n   at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext context)\n   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest& request)\n   at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest& request)\n   at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest& request)\n   at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)\n   at Autofac.Core.Container.ResolveComponent(ResolveRequest& request)\n   at Autofac.Core.Container.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request)\n   at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)\n   at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)\n   at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters)\n   at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)\n   at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)\n   at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)\n   at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.ReflectionMiddlewareBinder.CreateMiddleware(RequestDelegate next)\n   at Microsoft.AspNetCore.Builder.ApplicationBuilder.Build()\n   at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication()\n   at Microsoft.AspNetCore.Hosting.WebHost.StartAsync(CancellationToken cancellationToken)\n   at Microsoft.AspNetCore.Hosting.WebHost.Start()\n   at Seq.ServiceProcess.ServerService.Start()\n   at Seq.Cli.RunCommand.RunStandardIOAsync(Func`1 bootstrap, TextWriter cout)"}
{"@t":"2024-05-22T13:30:52.4957751Z","@mt":"Seq {SeqVersion} running on OS {OSCaption}","SeqVersion":"2024.3.11510","OSCaption":"Ubuntu 22.04.4 LTS","SourceContext":"Seq.Server.Features.Diagnostics.EnvironmentInformationWriter"}
{"@t":"2024-05-22T13:30:53.1751655Z","@mt":"Opening event store at {StreamPath}","StreamPath":"/data/Stream/stream.flare","SourceContext":"Seq.Engine.Cluster.StorageNodeCoordinator"}
{"@t":"2024-05-22T13:30:53.4954710Z","@mt":"Ingestion enabled","SourceContext":"Seq.Engine.Cluster.StorageNodeCoordinator"}
{"@t":"2024-05-22T13:30:53.5296917Z","@mt":"Opening metastore {MetastorePath}","MetastorePath":"/data/Documents/metastore.flare","SourceContext":"Quince.Storage.Native.NativeDocumentStore"}
{"@t":"2024-05-22T13:30:53.7557537Z","@mt":"Closing event store","SourceContext":"Seq.Engine.Cluster.StorageNodeCoordinator"}

[nblumhardt]

Thanks for that. My best guess is that your backing storage isn't the Premium version of Azure Files? This kind of corruption is common with the non-Premium Azure Files variant. Interesting to know, either way :-)

[rodneytamblyn]

It is premium storage. I think the issue might have been that a second container instance that I was trying to deploy started before the first one fully exited. That's my guess. Is it safe to use container stop for a safe shutdown or is there some command I can execute against the container for a safe orderly shutdown:

e.g. using AZ commands something similar to

az container exec --resource-group MYGROUP --name <container-name> --exec-command "<command>"

[rodneytamblyn]

FYI Our Azure Container INstance deployment of Seq has been going well over the last two months. I wanted to redeploy it to add the seq-input-syslog sidecar container, and the only way I can do this is to stop and delete the current instance and replace it with a new instance. This was when the corruption happened. Makes me nervous about upgrading the container.