A TCP socket is a specific TCP port on a specific node. Port exhaustion occurs when a node runs out of available ports. When an application stops using a specific port, the port enters a "time-wait state" before it becomes available for use by another application. If a node runs out of available ports to create new connections, the symptom appears as a spike in application errors.  

Application Symptoms

A DevOps engineer is responsible for monitoring the performance of a mission-critical app. She opens the Application Dashboard and notices a sudden spike in transaction errors on the Ecom-Tiers and for the entire application. She also sees that response times for the entire App are going down and load is increasing. She decides to investigate.

Application Flow Map

Network Diagnosis

  1. She switches over to the Network Dashboard, right-clicks Ecom-Tier2, and selects View Metrics. The Host Stack KPIs show an increase in wait sockets at the same time as the increase in Error transactions.
     Network impact on transactions
    Host Stack KPIs
  2. She knows there are many wait sockets for the entire tier. She wants to pinpoint the specific node (or nodes) where these wait sockets are occurring. She returns to the Network Dashboard, clicks Ecom-Tier2, and clicks ECOM_T2N1 in the Nodes tab.
    Nodes tab 
  3. The TCP Socket chart (right) for node ECOM_T2N2 shows that the level of wait sockets is in an acceptable range. She switches over to view ECOM_T2N1.
    TCP Socket chart 
  4. The TCP Socket chart for this node shows a significantly high number of wait sockets. She knows now that the spike in application errors is occurring because this node is running out of available sockets.