How to make Iot Edge message routing between modules work on arm32 (Raspberry Pi) - raspberry-pi

I created an Iot Edge solution using Visual Studio Code via View -> Command Palette... -> AzureIotEdge:NewIoTEdgeSolution using the C# Module Module Template. This generates an Iot Edge solution with two Modules: tempSensor creates messages with dummy telemetry-data and SampleModule is supposed to receive those messages, print them and then pass them on to IotHub.
I build the solution for arm32 and deployed it to a Rasperry Pi (running latest Raspbian). The modules are then displayed as running on the device but if I monitor d2c-messages there are none arriving in IoT Hub.
If I run sudo iotedge logs tempSensor -f on the device, I can see messages are being generated.
If I run sudo iotedge logs SampleModule -f I only see IoT Hub module client initialized. Which is logged in the Init() Method. There is no logging from the PipeMessage Method which seems not to be reached at all.
If I build the solution for amd64 and deploy it to a virtual machine, everything is working as expected.
Is there a way to get it working on a Raspberry Pi running Raspian?
By the way: I tried the Azure Functions C# Module Template as well. Same result: On amd64 it works but on arm32 it does not.
UPDATE:
Here is the logging of edgeAgent:
█████╗ ███████╗██╗ ██╗██████╗ ███████╗
██╔══██╗╚══███╔╝██║ ██║██╔══██╗██╔════╝
███████║ ███╔╝ ██║ ██║██████╔╝█████╗
██╔══██║ ███╔╝ ██║ ██║██╔══██╗██╔══╝
██║ ██║███████╗╚██████╔╝██║ ██║███████╗
╚═╝ ╚═╝╚══════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝
██╗ ██████╗ ████████╗ ███████╗██████╗ ██████╗ ███████╗
██║██╔═══██╗╚══██╔══╝ ██╔════╝██╔══██╗██╔════╝ ██╔════╝
██║██║ ██║ ██║ █████╗ ██║ ██║██║ ███╗█████╗
██║██║ ██║ ██║ ██╔══╝ ██║ ██║██║ ██║██╔══╝
██║╚██████╔╝ ██║ ███████╗██████╔╝╚██████╔╝███████╗
╚═╝ ╚═════╝ ╚═╝ ╚══════╝╚═════╝ ╚═════╝ ╚══════╝
2018-12-21 07:08:27.323 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
2018-12-21 07:08:31.986 +00:00 [INF] - Created persistent store at /tmp/edgeAgent
2018-12-21 07:08:34.193 +00:00 [INF] - Edge agent connected to IoT Hub via Amqp_Tcp_Only.
2018-12-21 07:08:37.624 +00:00 [INF] - Plan execution started for deployment 8
2018-12-21 07:08:37.654 +00:00 [INF] - Executing command: "Stop module SampleModule"
2018-12-21 07:08:37.766 +00:00 [INF] - Executing command: "Remove module SampleModule"
2018-12-21 07:08:37.964 +00:00 [INF] - Executing command: "Saving SampleModule to store"
2018-12-21 07:08:38.141 +00:00 [INF] - Executing command: "Command Group: (
[Create module SampleModule2]
[Start module SampleModule2]
)"
2018-12-21 07:08:38.152 +00:00 [INF] - Executing command: "Create module SampleModule2"
2018-12-21 07:08:42.678 +00:00 [INF] - Executing command: "Start module SampleModule2"
2018-12-21 07:08:45.749 +00:00 [INF] - Executing command: "Command Group: (
[Stop module tempSensor]
[Start module tempSensor]
[Saving tempSensor to store]
)"
2018-12-21 07:08:45.750 +00:00 [INF] - Executing command: "Stop module tempSensor"
2018-12-21 07:08:45.759 +00:00 [INF] - Executing command: "Start module tempSensor"
2018-12-21 07:08:48.423 +00:00 [INF] - Executing command: "Saving tempSensor to store"
2018-12-21 07:08:48.491 +00:00 [INF] - Executing command: "Command Group: (
[Stop module edgeHub]
[Start module edgeHub]
[Saving edgeHub to store]
)"
2018-12-21 07:08:48.491 +00:00 [INF] - Executing command: "Stop module edgeHub"
2018-12-21 07:08:48.501 +00:00 [INF] - Executing command: "Start module edgeHub"
2018-12-21 07:08:51.895 +00:00 [INF] - Executing command: "Saving edgeHub to store"
2018-12-21 07:08:51.900 +00:00 [INF] - Plan execution ended for deployment 8
2018-12-21 07:08:52.743 +00:00 [INF] - Updated reported properties
2018-12-21 07:08:57.811 +00:00 [INF] - HealthRestartPlanner is clearing restart stats for module 'tempSensor' as it has been running healthy for 00:10:00.
2018-12-21 07:08:57.814 +00:00 [INF] - HealthRestartPlanner is clearing restart stats for module 'edgeHub' as it has been running healthy for 00:10:00.
2018-12-21 07:08:57.815 +00:00 [INF] - Plan execution started for deployment 8
2018-12-21 07:08:57.818 +00:00 [INF] - Executing command: "Saving tempSensor to store"
2018-12-21 07:08:57.819 +00:00 [INF] - Executing command: "Saving edgeHub to store"
2018-12-21 07:08:57.820 +00:00 [INF] - Plan execution ended for deployment 8
2018-12-21 07:08:58.038 +00:00 [INF] - Updated reported properties
2018-12-21 07:09:03.239 +00:00 [INF] - Updated reported properties
And here some of the logging of edgeHub (there is lots more but seems to be all the same Exceptions):
2018-12-21 07:28:23.450 +00:00 [WRN] - Error sending messages to module bewaPi1/SampleModule2
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at DotNetty.Buffers.HeapArena.NewChunk(Int32 pageSize, Int32 maxOrder, Int32 pageShifts, Int32 chunkSize)
at DotNetty.Buffers.PoolArena`1.AllocateNormal(PooledByteBuffer`1 buf, Int32 reqCapacity, Int32 normCapacity)
at DotNetty.Buffers.PoolArena`1.Allocate(PoolThreadCache`1 cache, PooledByteBuffer`1 buf, Int32 reqCapacity)
at DotNetty.Buffers.PooledByteBufferAllocator.NewHeapBuffer(Int32 initialCapacity, Int32 maxCapacity)
at DotNetty.Buffers.AbstractByteBufferAllocator.Buffer(Int32 initialCapacity, Int32 maxCapacity)
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ByteBufferConverter.ToByteBuffer(Byte[] bytes) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ByteBufferConverter.cs:line 39
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ProtocolGatewayMessageConverter.FromMessage(IMessage message) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ProtocolGatewayMessageConverter.cs:line 101
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.DeviceProxy.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/DeviceProxy.cs:line 73
at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 256
at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.<>c__DisplayClass5_2.<<ProcessAsync>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 98
2018-12-21 07:28:28.517 +00:00 [WRN] - Non retryable exception occurred while sending message.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at DotNetty.Buffers.HeapArena.NewChunk(Int32 pageSize, Int32 maxOrder, Int32 pageShifts, Int32 chunkSize)
at DotNetty.Buffers.PoolArena`1.AllocateNormal(PooledByteBuffer`1 buf, Int32 reqCapacity, Int32 normCapacity)
at DotNetty.Buffers.PoolArena`1.Allocate(PoolThreadCache`1 cache, PooledByteBuffer`1 buf, Int32 reqCapacity)
at DotNetty.Buffers.PooledByteBufferAllocator.NewHeapBuffer(Int32 initialCapacity, Int32 maxCapacity)
at DotNetty.Buffers.AbstractByteBufferAllocator.Buffer(Int32 initialCapacity, Int32 maxCapacity)
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ByteBufferConverter.ToByteBuffer(Byte[] bytes) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ByteBufferConverter.cs:line 39
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ProtocolGatewayMessageConverter.FromMessage(IMessage message) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ProtocolGatewayMessageConverter.cs:line 101
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.DeviceProxy.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/DeviceProxy.cs:line 73
at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 256
at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.<>c__DisplayClass5_2.<<ProcessAsync>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 98
2018-12-21 07:28:28.521 +00:00 [WRN] - Error sending messages to module bewaPi1/SampleModule2
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at DotNetty.Buffers.HeapArena.NewChunk(Int32 pageSize, Int32 maxOrder, Int32 pageShifts, Int32 chunkSize)
at DotNetty.Buffers.PoolArena`1.AllocateNormal(PooledByteBuffer`1 buf, Int32 reqCapacity, Int32 normCapacity)
at DotNetty.Buffers.PoolArena`1.Allocate(PoolThreadCache`1 cache, PooledByteBuffer`1 buf, Int32 reqCapacity)
at DotNetty.Buffers.PooledByteBufferAllocator.NewHeapBuffer(Int32 initialCapacity, Int32 maxCapacity)
at DotNetty.Buffers.AbstractByteBufferAllocator.Buffer(Int32 initialCapacity, Int32 maxCapacity)
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ByteBufferConverter.ToByteBuffer(Byte[] bytes) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ByteBufferConverter.cs:line 39
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ProtocolGatewayMessageConverter.FromMessage(IMessage message) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ProtocolGatewayMessageConverter.cs:line 101
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.DeviceProxy.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/DeviceProxy.cs:line 73
at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 256
at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.<>c__DisplayClass5_2.<<ProcessAsync>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 98
2018-12-21 07:28:33.666 +00:00 [WRN] - Non retryable exception occurred while sending message.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at DotNetty.Buffers.HeapArena.NewChunk(Int32 pageSize, Int32 maxOrder, Int32 pageShifts, Int32 chunkSize)
at DotNetty.Buffers.PoolArena`1.AllocateNormal(PooledByteBuffer`1 buf, Int32 reqCapacity, Int32 normCapacity)
at DotNetty.Buffers.PoolArena`1.Allocate(PoolThreadCache`1 cache, PooledByteBuffer`1 buf, Int32 reqCapacity)
at DotNetty.Buffers.PooledByteBufferAllocator.NewHeapBuffer(Int32 initialCapacity, Int32 maxCapacity)
at DotNetty.Buffers.AbstractByteBufferAllocator.Buffer(Int32 initialCapacity, Int32 maxCapacity)
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ByteBufferConverter.ToByteBuffer(Byte[] bytes) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ByteBufferConverter.cs:line 39
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ProtocolGatewayMessageConverter.FromMessage(IMessage message) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ProtocolGatewayMessageConverter.cs:line 101
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.DeviceProxy.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/DeviceProxy.cs:line 73
at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 256
at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.<>c__DisplayClass5_2.<<ProcessAsync>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 98
2018-12-21 07:28:33.669 +00:00 [WRN] - Error sending messages to module bewaPi1/SampleModule2
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at DotNetty.Buffers.HeapArena.NewChunk(Int32 pageSize, Int32 maxOrder, Int32 pageShifts, Int32 chunkSize)
at DotNetty.Buffers.PoolArena`1.AllocateNormal(PooledByteBuffer`1 buf, Int32 reqCapacity, Int32 normCapacity)
at DotNetty.Buffers.PoolArena`1.Allocate(PoolThreadCache`1 cache, PooledByteBuffer`1 buf, Int32 reqCapacity)
at DotNetty.Buffers.PooledByteBufferAllocator.NewHeapBuffer(Int32 initialCapacity, Int32 maxCapacity)
at DotNetty.Buffers.AbstractByteBufferAllocator.Buffer(Int32 initialCapacity, Int32 maxCapacity)
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ByteBufferConverter.ToByteBuffer(Byte[] bytes) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ByteBufferConverter.cs:line 39
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ProtocolGatewayMessageConverter.FromMessage(IMessage message) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ProtocolGatewayMessageConverter.cs:line 101
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.DeviceProxy.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/DeviceProxy.cs:line 73
at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 256
at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.<>c__DisplayClass5_2.<<ProcessAsync>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 98
2018-12-21 07:28:39.012 +00:00 [WRN] - Non retryable exception occurred while sending message.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at DotNetty.Buffers.HeapArena.NewChunk(Int32 pageSize, Int32 maxOrder, Int32 pageShifts, Int32 chunkSize)
at DotNetty.Buffers.PoolArena`1.AllocateNormal(PooledByteBuffer`1 buf, Int32 reqCapacity, Int32 normCapacity)
at DotNetty.Buffers.PoolArena`1.Allocate(PoolThreadCache`1 cache, PooledByteBuffer`1 buf, Int32 reqCapacity)
at DotNetty.Buffers.PooledByteBufferAllocator.NewHeapBuffer(Int32 initialCapacity, Int32 maxCapacity)
at DotNetty.Buffers.AbstractByteBufferAllocator.Buffer(Int32 initialCapacity, Int32 maxCapacity)
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ByteBufferConverter.ToByteBuffer(Byte[] bytes) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ByteBufferConverter.cs:line 39
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ProtocolGatewayMessageConverter.FromMessage(IMessage message) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ProtocolGatewayMessageConverter.cs:line 101
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.DeviceProxy.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/DeviceProxy.cs:line 73
at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 256
at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.<>c__DisplayClass5_2.<<ProcessAsync>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 98
2018-12-21 07:28:39.015 +00:00 [WRN] - Error sending messages to module bewaPi1/SampleModule2
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at DotNetty.Buffers.HeapArena.NewChunk(Int32 pageSize, Int32 maxOrder, Int32 pageShifts, Int32 chunkSize)
at DotNetty.Buffers.PoolArena`1.AllocateNormal(PooledByteBuffer`1 buf, Int32 reqCapacity, Int32 normCapacity)
at DotNetty.Buffers.PoolArena`1.Allocate(PoolThreadCache`1 cache, PooledByteBuffer`1 buf, Int32 reqCapacity)
at DotNetty.Buffers.PooledByteBufferAllocator.NewHeapBuffer(Int32 initialCapacity, Int32 maxCapacity)
at DotNetty.Buffers.AbstractByteBufferAllocator.Buffer(Int32 initialCapacity, Int32 maxCapacity)
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ByteBufferConverter.ToByteBuffer(Byte[] bytes) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ByteBufferConverter.cs:line 39
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ProtocolGatewayMessageConverter.FromMessage(IMessage message) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ProtocolGatewayMessageConverter.cs:line 101
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.DeviceProxy.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/DeviceProxy.cs:line 73
at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 256
at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.<>c__DisplayClass5_2.<<ProcessAsync>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 98
So the problem is caused by System.OutOfMemoryExceptions. But what is causing those exceptions?
Update2: This seems to be a known problem (see here) and can be solved by Setting OptimizeForPerformance to false. I will try this and see if it works.

The message routing between modules does not work because of System.OutOfMemoryExceptions in the edgeHub module. The problem is described and discussed here.
Microsoft has implemented a solution which is described here.
In short: If running IotEdge Solutions on Raspberry Pi the environment variable OptimizeForPerformance has to be set to false.

Seems the issue related to
https://learn.microsoft.com/en-us/azure/iot-edge/troubleshoot#stability-issues-on-resource-constrained-devices
Please refer the link and turn off OptimizeForPerformance in deployment manifest.

Related

TEIID40007 Keepalive failed for session

I am trying to connect to federated vdb via Spotfire. I keep getting following error after fetching 200K plus records. There total of 3M records in the view
TEIID session timeout has no limit.
TEIID40007 Keepalive failed for session E0NBLogYqzk3
I am not experiencing the above error when I try to read the vdb in sql client like dbeaver or squirrel and can fetch entire dataset.
Below is the snippet from server.log in Jboss:
setup [PolicyOutInterceptor]
pre-logical [ClientRequestFilterInterceptor]
prepare-send [MessageSenderInterceptor]
write [BodyWriter]
prepare-send-ending [MessageSenderEndingInterceptor]
2022-09-08 12:01:06,358 FINE [org.apache.cxf.phase.PhaseInterceptorChain] (Worker1579_QueryProcessorQueue29181462) Invoking handleMessage on interceptor org.apache.cxf.jaxrs.client.WebClient$BodyWriter#511fa86c
2022-09-08 12:01:06,358 FINE [org.apache.cxf.phase.PhaseInterceptorChain] (Worker1579_QueryProcessorQueue29181462) Invoking handleMessage on interceptor org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor#4a217e2f
2022-09-08 12:01:06,359 FINE [org.apache.cxf.transport.http.Headers] (Worker1579_QueryProcessorQueue29181462) Accept: application/json
2022-09-08 12:01:06,359 FINE [org.apache.cxf.transport.http.Headers] (Worker1579_QueryProcessorQueue29181462) User-Agent: Teiid Server
2022-09-08 12:01:06,359 FINE [org.apache.cxf.transport.http.Headers] (Worker1579_QueryProcessorQueue29181462) Content-length: 0
2022-09-08 12:01:06,359 FINE [org.apache.cxf.transport.http.Headers] (Worker1579_QueryProcessorQueue29181462) Content-Type: application/json
2022-09-08 12:01:06,359 FINE [org.apache.cxf.transport.http.HTTPConduit] (Worker1579_QueryProcessorQueue29181462) No Trust Decider for Conduit '{http://solrserver.net:8983/solr/document/select}WebClient.http-conduit'. An afirmative Trust Decision is assumed.
2022-09-08 12:01:06,359 FINE [org.apache.cxf.transport.http.HTTPConduit] (Worker1579_QueryProcessorQueue29181462) Sending POST Message with Headers to http://solrserver.net:8983/solr/document/select Conduit :{http://solrserver.net:8983/solr/document/select}WebClient.http-conduit
2022-09-08 12:01:06,359 INFO [org.teiid.SECURITY] (SessionMonitor) TEIID40007 Keepalive failed for session E0NBLogYqzk3
2022-09-08 12:01:06,359 INFO [org.teiid.SECURITY] (SessionMonitor) TEIID40007 Keepalive failed for session E0NBLogYqzk3
2022-09-08 12:01:06,359 DEBUG [org.teiid.SECURITY] (SessionMonitor) closeSession E0NBLogYqzk3
2022-09-08 12:01:06,359 DEBUG [org.teiid.SECURITY] (SessionMonitor) closeSession E0NBLogYqzk3
2022-09-08 12:01:06,359 DEBUG [org.teiid.AUDIT_LOG] (SessionMonitor) [svcpds] <session.logoff>
2022-09-08 12:01:06,359 DEBUG [org.teiid.AUDIT_LOG] (SessionMonitor) [svcpds] <session.logoff>
2022-09-08 12:01:06,359 DEBUG [org.teiid.COMMAND_LOG] (SessionMonitor) CANCEL SRC COMMAND: endTime=2022-09-08 12:01:06.359 requestID=E0NBLogYqzk3.5 sourceCommandID=7 executionID=8950651 txID=null modelName=PDS_SOURCE_MODEL translatorName=delegate sessionID=E0NBLogYqzk3 principal=svcpds finalRowCount=-1
2022-09-08 12:01:06,359 DEBUG [org.teiid.COMMAND_LOG] (SessionMonitor) CANCEL SRC COMMAND: endTime=2022-09-08 12:01:06.359 requestID=E0NBLogYqzk3.5 sourceCommandID=7 executionID=8950651 txID=null modelName=PDS_SOURCE_MODEL translatorName=delegate sessionID=E0NBLogYqzk3 principal=svcpds finalRowCount=-1
2022-09-08 12:01:06,360 FINE [org.apache.cxf.phase.PhaseInterceptorChain] (Worker1579_QueryProcessorQueue29181462) Adding interceptor org.apache.cxf.ws.policy.PolicyInInterceptor#3404a1c0 to phase receive
2022-09-08 12:01:06,360 FINE [org.apache.cxf.phase.PhaseInterceptorChain] (Worker1579_QueryProcessorQueue29181462) Adding interceptor org.apache.cxf.jaxrs.client.WebClient$ClientAsyncResponseInterceptor#6f403813 to phase unmarshal
2022-09-08 12:01:06,360 DEBUG [org.teiid.COMMAND_LOG] (SessionMonitor) CANCEL SRC COMMAND: endTime=2022-09-08 12:01:06.36 requestID=E0NBLogYqzk3.5 sourceCommandID=3 executionID=9226341 txID=null modelName=GF translatorName=rest sessionID=E0NBLogYqzk3 principal=svcpds finalRowCount=-1
2022-09-08 12:01:06,360 FINE [org.apache.cxf.phase.PhaseInterceptorChain] (Worker1579_QueryProcessorQueue29181462) Adding interceptor org.apache.cxf.jaxrs.client.spec.ClientResponseFilterInterceptor#43b0a15f to phase pre-protocol-frontend
2022-09-08 12:01:06,360 FINE [org.apache.cxf.phase.PhaseInterceptorChain] (Worker1579_QueryProcessorQueue29181462) Chain org.apache.cxf.phase.PhaseInterceptorChain#5af24963 was created. Current flow:
receive [PolicyInInterceptor]
pre-protocol-frontend [ClientResponseFilterInterceptor]
unmarshal [ClientAsyncResponseInterceptor]
Below is the error I am getting on Spotfire client:
ImportException at Spotfire.Dxp.Data:
Failed to create DataTable (HRESULT: 80131500)
Stack Trace:
at Spotfire.Dxp.Data.ColumnFactory.CreateColumns(DataRowReader reader, String documentTitleForOrigin, IDataPropertyContainer defaultProperties, DataPropertyRegistry dataPropertyRegistry, GlobalMethodRegistry globalMethodRegistry, CxxSession session, Boolean addNewProperties, PartialDataLoadReport loadReport, ResultProperties resultProperties, PendingViewRequestsManager pendingViewRequestsManager, Boolean mangleColumnNames)
at Spotfire.Dxp.Data.Producers.SourceColumnProducer.<>c__DisplayClass75_0.<CreateView>b__0()
at Spotfire.Dxp.Framework.ApplicationModel.Progress.ExecuteSubtask(String title, ProgressOperation operation)
at Spotfire.Dxp.Data.Producers.SourceColumnProducer.CreateView(CxxSession session, DataPropertyRegistry propertyRegistry, GlobalMethodRegistry globalMethodRegistry, DataSourceConnection connection, IDataPropertyContainer defaultColumnProperties, PartialDataLoadReport& partialLoadReport)
at Spotfire.Dxp.Data.Producers.SourceColumnProducer.GetColumnsAndProperties(DataSourceConnection connection)
at Spotfire.Dxp.Data.Persistence.DataItem.PerformUpdate(SourceColumnProducer producer, DataSourceConnection connection)
at Spotfire.Dxp.Data.Persistence.DataItem.Update(SourceColumnProducer producer, DataSourceConnection connection)
at Spotfire.Dxp.Data.Persistence.DataPool.<LoadData>d__15.MoveNext()
at Spotfire.Dxp.Data.Producers.SourceColumnProducer.OnConfigure()
at Spotfire.Dxp.Framework.DocumentModel.Node.ConfigureSubTree()
at Spotfire.Dxp.Framework.DocumentModel.Node.<>c.<ConfigureSubTree>b__47_0(Node node)
at Spotfire.Dxp.Framework.DocumentModel.UndoableListAvlLeaf`1.ForEachChild(Action`1 action, Boolean includeFrozen)
at Spotfire.Dxp.Framework.DocumentModel.Node.ConfigureSubTree()
at Spotfire.Dxp.Framework.DocumentModel.Node.<>c.<ConfigureSubTree>b__47_0(Node node)
at Spotfire.Dxp.Framework.DocumentModel.UndoableList`1.ForEachChild(Action`1 action, Boolean includeFrozen)
at Spotfire.Dxp.Framework.DocumentModel.Node.ConfigureSubTree()
at Spotfire.Dxp.Framework.DocumentModel.Node.<>c.<ConfigureSubTree>b__47_0(Node node)
at Spotfire.Dxp.Framework.DocumentModel.State.NodeState.<>c__DisplayClass92_0.<ForEachManagedChild>b__0(IDocumentNodeChild documentNodeChild)
at Spotfire.Dxp.Framework.DocumentModel.State.NodeState.ForEachChild(IDocumentNodeChild[] children, Action`1 action)
at Spotfire.Dxp.Framework.DocumentModel.Node.ConfigureSubTree()
at Spotfire.Dxp.Framework.ApplicationModel.Progress.ExecuteSubtask(String title, IndeterminateProgressFormatter progressFormatter, ProgressOperation operation)
at Spotfire.Dxp.Framework.DocumentModel.DocumentNode.ConfigureAndAttachFromNew()
at Spotfire.Dxp.Framework.DocumentModel.DocumentNode.AttachSubTreeWhileExecuting(UndoableNodeBase newOwner)
at Spotfire.Dxp.Framework.DocumentModel.UndoableNode.Spotfire.Dxp.Framework.DocumentModel.IUndoableNode.AttachItemToUndoableNode(Object item)
at Spotfire.Dxp.Framework.DocumentModel.UndoableKeyedCollection`2.<>c__DisplayClass43_0.<Insert>b__0()
at Spotfire.Dxp.Framework.DocumentModel.Node.InternalTransaction(Executor executor, Boolean rollbackNestedInternalTransactionAtException, Boolean isStreamingProperty)
at Spotfire.Dxp.Framework.DocumentModel.UndoableKeyedCollection`2.Insert(Int32 index, TNode item)
at Spotfire.Dxp.Data.DataTableCollection.<>c__DisplayClass89_0.<Add>b__0()
at Spotfire.Dxp.Framework.DocumentModel.Node.InternalTransaction(Executor executor, Boolean rollbackNestedInternalTransactionAtException, Boolean isStreamingProperty)
at Spotfire.Dxp.Data.DataTableCollection.Add(DataTable dataTable)
at Spotfire.Dxp.Application.PartiallyOpenedDataSource.<LoadData>d__9.MoveNext()
at Spotfire.Dxp.Application.AnalysisApplication.<OpenPartiallyOpenedDocument>d__98.MoveNext()
at Spotfire.Dxp.Application.AnalysisApplication.<OpenDataSource>d__91.MoveNext()
at Spotfire.Dxp.Application.AnalysisApplication.ConsumeDataLoadPromptRequests(IEnumerable`1 prompts)
at Spotfire.Dxp.Application.AnalysisApplication.Open(DataSource source, DocumentOpenSettings settings)
at Spotfire.Dxp.Forms.Data.Import.DataSourceFactoryService.OpenDataSourceWithoutPrompting(DataSource dataSource, DocumentOpenSettings documentOpenSettings, IServiceProvider serviceProvider)
at Spotfire.Dxp.Forms.Data.Import.DataSourceFactoryService.OpenDataSource(InformationLinkDataSource dataSource, DocumentOpenSettings documentOpenSettings, IServiceProvider serviceProvider)
at Spotfire.Dxp.Framework.ApplicationModel.Progress.<>c__DisplayClass21_0.<Start>b__0()
at Spotfire.Dxp.Framework.ApplicationModel.MonitorableProgress.Start[T](Func`1 action)
at Spotfire.Dxp.Forms.Application.FormsProgressService.ProgressThread.DoOperationLoop()
InformationModelException at Spotfire.Dxp.Data:
Failed to get data: 57014 TEIID30160 The request /IK4StVK70Kl.1 has been cancelled. (HRESULT: 80131500)
Stack Trace:
at Spotfire.Dxp.Data.InformationModel.InternalInformationModelManager.DataStream.GetNextBlock()
at Spotfire.Dxp.Data.InformationModel.InternalInformationModelManager.DataStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at Spotfire.Dxp.Internal.Utilities.SeekableStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at Spotfire.Dxp.Framework.ApplicationModel.ProgressIncrementStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at Spotfire.Dxp.Internal.Utilities.SharedMemoryStream.CopyUnprotected(Stream inputStream, Int32 bufferSize)
at Spotfire.Dxp.Data.Cxx.CxxColumnManager.LoadSBDF(String id, Stream shm, Action streamWriter, Action abortAction, PartialTableCallback partialTableCallback, Boolean useFasterWipImplementation)
at Spotfire.Dxp.Data.Cxx.CxxDataTransfer.LoadSBDF(CxxSession session, Stream stream, PartialTableCallback partialTableCallback, Boolean useFasterSBDF)
at Spotfire.Dxp.Data.Import.SbdfDataRowReader.TryCreateCxxRepresentation(CxxSession session, Int64 exclusiveStartRowIndex, UInt64 maxRowsThatWillBeRead, PendingViewRequestsManager pendingViewRequestsManager, CxxTable& table)
at Spotfire.Dxp.Data.Cxx.CxxDataTransfer.CreateTable(DataRowReader dataRowReader, CxxSession session, PartialDataLoadReport report, Advancer rowAdvancer, Boolean needsReset, Int64 exclusiveStartRowIndex, UInt64 maxRowsThatWillBeRead, PendingViewRequestsManager partiallyLoadedConsumers)
at Spotfire.Dxp.Data.Cxx.CxxDataTransfer.CreateTable(DataRowReader dataRowReader, CxxSession session, PartialDataLoadReport report, PendingViewRequestsManager pendingViewRequestsManager, Boolean needsReset, UInt64 maxRowsToConsume)
at Spotfire.Dxp.Data.ColumnFactory.CreateColumns(DataRowReader reader, String documentTitleForOrigin, IDataPropertyContainer defaultProperties, DataPropertyRegistry dataPropertyRegistry, GlobalMethodRegistry globalMethodRegistry, CxxSession session, Boolean addNewProperties, PartialDataLoadReport loadReport, ResultProperties resultProperties, PendingViewRequestsManager pendingViewRequestsManager, Boolean mangleColumnNames)
InformationModelServiceException at Spotfire.Dxp.Services:
Failed to get data: 57014 TEIID30160 The request /IK4StVK70Kl.1 has been cancelled. (HRESULT: 80131509)
Stack Trace:
at Spotfire.Dxp.Services.WebServiceBase`1.InvokeService[T](ServiceMethod`1 serviceMethod, ExceptionFactoryMethod exceptionFactoryMethod, String customMethodNameForLogging)
at Spotfire.Dxp.Services.Data.InformationModel.QueryManagerService.InvokeService[T](ServiceMethod`1 serviceMethod)
at Spotfire.Dxp.Data.InformationModel.InternalInformationModelManager.DataStream.GetNextBlock()
Appreciate the help. Thanks

Debug "An item with the same key has already been added" exception

I took a crash dump of my application when I got the "An item with the same key has already been added" exception. I need help finding which object caused this exception. I could print the exception but couldn't figure out how to find the exact key that caused the exception.
This is likely the state you have:
[...]
(3250.7ec): CLR exception - code e0434352 (!!! second chance !!!)
[...]
0:000> .loadby sos clr
0:000> !pe
Exception object: 030c31e8
Exception type: System.ArgumentException
Message: An item with the same key has already been added.
InnerException: <none>
StackTrace (generated):
SP IP Function
010FEE1C 6045F705 mscorlib_ni!System.ThrowHelper.ThrowArgumentException(System.ExceptionResource)+0x35
010FEE2C 609410C7 mscorlib_ni!System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Insert(System.__Canon, System.__Canon, Boolean)+0xc6af67
010FEE60 5FD4B310 mscorlib_ni!System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Add(System.__Canon, System.__Canon)+0x10
010FEE68 017004F5 KeyAlreadyAdded!KeyAlreadyAdded.Program.Main()+0x45
[...]
In the native call stack, you can see the call to Dictionary.Add() again, but with the additional information for the frame number:
0:000> k
# ChildEBP RetAddr
00 010fecb0 618fac03 KERNELBASE!RaiseException+0x62
01 010fed4c 618fae08 clr!RaiseTheExceptionInternalOnly+0x27c
02 010fee14 6045f705 clr!IL_Throw+0x141
03 010fee24 609410c7 mscorlib_ni!System.ThrowHelper.ThrowArgumentException(System.ExceptionResource)$##6000335+0x35
04 010fee50 5fd4b310 mscorlib_ni![COLD] System.Collections.Generic.Dictionary`2[System.__Canon,System.__Canon].Insert(System.__Canon, System.__Canon, Boolean)$##6003922+0x87
05 010fee68 6181ebe6 mscorlib_ni!System.Collections.Generic.Dictionary`2[System.__Canon,System.__Canon].Add(System.__Canon, System.__Canon)$##6003915+0x10
[...]
At the Insert() method, you can use the ebx register to get the key:
0:000> .frame /r 4
04 010fee50 5fd4b310 mscorlib_ni![COLD] System.Collections.Generic.Dictionary`2[System.__Canon,System.__Canon].Insert(System.__Canon, System.__Canon, Boolean)$##6003922+0x87
eax=010fec58 ebx=030c2364 ecx=00000005 edx=00000000 esi=030c23b0 edi=030c2364
[...]
0:000> !do 030c2364
Name: System.String
MethodTable: 5fdefd60
EEClass: 5f9c4e90
Size: 22(0x16) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String: this
[...]
So in this case, the duplicate key being added is the string "this". Here's the code:
using System.Collections.Generic;
namespace KeyAlreadyAdded
{
class Program
{
static Dictionary<string, string> dict = new Dictionary<string, string> {{"this", "was already inside"}};
static void Main()
{
dict.Add("that", "goes in easily");
dict.Add("this", "however, causes a duplicate key exception");
}
}
}

Downloading Cmake through Powershell script

I am trying to write a Powershell script for downloading the cmake zip file and installing it on a computer. While I thought it would be a simple task I am running into some issues with connecting to the cmake website. Here is my code:
$url = "https://cmake.org/files/v3.13/cmake-3.13.0-rc1-win64-x64.zip"
$output = "E:\test.zip"
$client = new-object System.Net.WebClient
$client.DownloadFile($url,$output)
When I run it, I get an error:
System.Management.Automation.MethodInvocationException: Exception calling "DownloadFile" with "2" argument(s): "The underlying connection was closed: An unexpected error occurred on a send." ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.WriteHeaders(Boolean async)
--- End of inner exception stack trace ---
at System.Net.WebClient.DownloadFile(Uri address, String fileName)
at CallSite.Target(Closure , CallSite , Object , Object , Object )
--- End of inner exception stack trace ---
at System.Management.Automation.ExceptionHandlingOps.ConvertToMethodInvocationException(Exception exception, Type typeToThrow, String methodName, Int32 numArgs, MemberInfo memberInfo)
at CallSite.Target(Closure , CallSite , Object , Object , Object )
at System.Management.Automation.Interpreter.DynamicInstruction`4.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
It looks like there is something with the cmake website that is closing the transport stream and preventing me from downloading the file. What should I do? How can I get around this?
Try adding the following lines preceding your code:
# Avoid errors with Invoke-WebRequest:
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12

Connect-AzureRmAccount : accessing_ws_metadata_exchange_failed

I get the following error when attempting to connect to Azure using PowerShell. This account previously worked, but not sure what changed. Multi-factor authentication was enabled for organization, but this account should be excluded. This is an organizational account; not a Microsoft account. I have found similar errors online, but not this particular issue...
Connect-AzureRmAccount –Credential $Credential -Verbose
VERBOSE: Performing the operation "log in" on target "User account in environment 'AzureCloud'".
Connect-AzureRmAccount : accessing_ws_metadata_exchange_failed: Accessing WS metadata exchange failed: The underlying connection was closed: An unexpected error
occurred on a send.
At line:1 char:1
+ Connect-AzureRmAccount –Credential $Credential -Verbose
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Connect-AzureRmAccount], AadAuthenticationFailedException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand
I enabled Debug output for more information...
DEBUG: AzureQoSEvent: CommandName - Connect-AzureRmAccount; IsSuccess - False; Duration - 00:00:19.9521148; Exception - Microsoft.Azure.Commands.Common.Authentica
tion.AadAuthenticationFailedException: accessing_ws_metadata_exchange_failed: Accessing WS metadata exchange failed: The underlying connection was closed: An unex
pected error occurred on a send. ---> Microsoft.IdentityModel.Clients.ActiveDirectory.AdalServiceException: accessing_ws_metadata_exchange_failed: Accessing WS me
tadata exchange failed ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Una
ble to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existin
g connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.WriteHeaders(Boolean async)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpWebRequestWrapper.<GetResponseSyncOrAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.IdentityModel.Clients.ActiveDirectory.MexParser.<FetchMexAsync>d__4.MoveNext()
--- End of inner exception stack trace ---
at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.RunAsyncTask[T](Task`1 task)
at Microsoft.Azure.Commands.Common.Authentication.UserTokenProvider.DoAcquireToken(AdalConfiguration config, PromptBehavior promptBehavior, Action`1 promptActi
on, String userId, SecureString password)
at Microsoft.Azure.Commands.Common.Authentication.UserTokenProvider.SafeAquireToken(AdalConfiguration config, String showDialog, Action`1 promptAction, String
userId, SecureString password, Exception& ex)
--- End of inner exception stack trace ---
at Microsoft.Azure.Commands.Common.Authentication.UserTokenProvider.AcquireToken(AdalConfiguration config, String promptBehavior, Action`1 promptAction, String
userId, SecureString password)
at Microsoft.Azure.Commands.Common.Authentication.UserTokenProvider.GetAccessToken(AdalConfiguration config, String promptBehavior, Action`1 promptAction, Stri
ng userId, SecureString password, String credentialType)
at Microsoft.Azure.Commands.Common.Authentication.Factories.AuthenticationFactory.Authenticate(IAzureAccount account, IAzureEnvironment environment, String ten
ant, SecureString password, String promptBehavior, Action`1 promptAction, IAzureTokenCache tokenCache, String resourceId)
at Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient.AcquireAccessToken(IAzureAccount account, IAzureEnvironment environment, String tenantId, Se
cureString password, String promptBehavior, Action`1 promptAction)
at Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient.ListAccountTenants(IAzureAccount account, IAzureEnvironment environment, SecureString passwo
rd, String promptBehavior, Action`1 promptAction)
at Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient.Login(IAzureAccount account, IAzureEnvironment environment, String tenantId, String subscrip
tionId, String subscriptionName, SecureString password, Boolean skipValidation, Action`1 promptAction, String name)
at Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand.<>c__DisplayClass83_0.<ExecuteCmdlet>b__0(AzureRmProfile localProfile, RMProfileClient profile
Client, String name)
at Microsoft.Azure.Commands.Profile.Common.AzureContextModificationCmdlet.ModifyContext(Action`2 contextAction)
at Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand.SetContextWithOverwritePrompt(Action`3 setContextAction)
at Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand.ExecuteCmdlet()
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord();
DEBUG: Finish sending metric.
DEBUG: 11:22:02 AM - ConnectAzureRmAccountCommand end processing.
DEBUG: 11:22:02 AM - ConnectAzureRmAccountCommand end processing.
The issue was that the Connect-AzureRMAccount cmdlet used TLS 1.0 by default. Centrify disabled TLS 1.0 support which caused the authentication attempts to close connection. The issue was resolved by adding the following to the PowerShell scripts to force TLS 1.2 security protocol:
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Sitecore 8 Analytics: PostSessionEndPipeline failed

I have recently configured MongoDB and Sitecore 8 Analytics (rev. 150812) on a production environment and it works just fine. However, oddly, I get this error in log files for every single pageview on live website, not sure why it exactly happens and how to rectify it. It is always a pair of errors (PostSessionEndPipeline failed + PostSessionEndPipeline failed). This happens on content delivery server. MongoDB is on another server. delivery server is on one Azure VM and MongoDB is on another VM. Does it have anything to do with session timeout or something like that?
7072 00:09:19 ERROR PostSessionEndPipeline failed.
Exception: System.IO.IOException
Message: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Source: System
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at MongoDB.Bson.IO.ByteBufferFactory.LoadFrom(Stream stream)
at MongoDB.Driver.Internal.MongoConnection.ReceiveMessage[TDocument](BsonBinaryReaderSettings readerSettings, IBsonSerializer serializer, IBsonSerializationOptions serializationOptions)
at MongoDB.Driver.Operations.QueryOperation`1.GetFirstBatch(IConnectionProvider connectionProvider)
at MongoDB.Driver.Operations.QueryOperation`1.Execute(IConnectionProvider connectionProvider)
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
at MongoDB.Driver.MongoCollection.FindOneAs[TDocument](FindOneArgs args)
at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbCollection.<>c__DisplayClass1d`1.<FindOneByIdAs>b__1c()
at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbCollection.Execute(Action action, ExceptionBehavior exceptionBehavior)
at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbCollection.FindOneByIdAs[TDocument](BsonValue value, Nullable`1 exceptionBehavior)
at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbDriver.FindOneByIdAs[T](Object key)
at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbDictionary.LoadAs[T](Object key)
at Sitecore.Analytics.Data.Dictionaries.TrackingDictionary`2.Get(TKey key, LookupStrategy strategy)
at Sitecore.Analytics.Pipelines.EnsureClassification.GetClassificators.Process(EnsureClassificationPipelineArgs args)
at (Object , Object[] )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at (Object , Object[] )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at Sitecore.Analytics.Pipelines.PostSessionEnd.CommitSession.Process(PostSessionEndArgs args)
at (Object , Object[] )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at Sitecore.Web.Application.RaiseSessionEndEvent(HttpApplication context)
Nested Exception
Exception: System.Net.Sockets.SocketException
Message: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
Source: System
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
1572 00:09:19 ERROR SessionEndPipeline failed.
Exception: System.IO.IOException
Message: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Source: System
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at MongoDB.Bson.IO.ByteBufferFactory.LoadFrom(Stream stream)
at MongoDB.Driver.Internal.MongoConnection.ReceiveMessage[TDocument](BsonBinaryReaderSettings readerSettings, IBsonSerializer serializer, IBsonSerializationOptions serializationOptions)
at MongoDB.Driver.Operations.QueryOperation`1.GetFirstBatch(IConnectionProvider connectionProvider)
at MongoDB.Driver.Operations.QueryOperation`1.Execute(IConnectionProvider connectionProvider)
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Sitecore.Analytics.Automation.Data.MongoDbAutomationProvider.GetAutomationDocuments(ID contactId, IReadOnlyCollection`1& stateDocuments, IReadOnlyCollection`1& transitionDocuments)
at Sitecore.Analytics.Automation.Data.MongoDbAutomationProvider.GetAutomationContext(ID contactId, DateTime timestampExclusive)
at Sitecore.Analytics.Automation.Data.AutomationProvider.GetAutomationContext(ID contactId)
at Sitecore.Analytics.Automation.Data.AutomationStateManager.Create(Contact contact)
at Sitecore.Analytics.Automation.ContactAutomationEvents.Run(Session session, PageEventItem pageEventDefinitionItem)
at (Object , Object[] )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at (Object , Object[] )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at Sitecore.Web.Application.RaiseSessionEndEvent(HttpApplication context)
Nested Exception
Exception: System.Net.Sockets.SocketException
Message: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
Source: System at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)