# Flink On Yarn Monitoring

> Measurement and monitoring of general metrics for Flink stream engine in Yarn running mode.

## Configuration Parameters

|   Parameter Name    |                                             Parameter Help Description                                              |
|---------------------|---------------------------------------------------------------------------------------------------------------------|
| Monitoring Host     | The monitored peer's IPV4, IPV6, or domain name. Note ⚠️ do not include protocol headers (e.g., https://, http://). |
| Task Name           | The name identifying this monitoring task. The name must be unique.                                                 |
| Yarn Port           | The Yarn port, corresponding to the port in `yarn.resourcemanager.webapp.address`.                                  |
| Query Timeout       | The timeout for JVM connections, in milliseconds, default is 3000 ms.                                               |
| Enable SSL          | Whether to enable SSL                                                                                               |
| Username            | Connection username                                                                                                 |
| Password            | Connection password                                                                                                 |
| Monitoring Interval | Interval for periodic data collection, in seconds, minimum interval is 30 seconds.                                  |
| Tags                | Used for categorizing and managing monitoring resources.                                                            |
| Description         | Additional notes and descriptions for this monitoring task. Users can add notes here.                               |

### Collected Metrics

#### Metrics Set: JobManager Metrics

|                      Metric Name                      | Metric Unit |              Metric Help Description               |
|-------------------------------------------------------|-------------|----------------------------------------------------|
| Status.JVM.Memory.NonHeap.Committed                   | Bytes       | Non-heap memory committed                          |
| Status.JVM.Memory.Mapped.TotalCapacity                | Bytes       | Total capacity of mapped memory                    |
| Status.JVM.Memory.NonHeap.Used                        | Bytes       | Non-heap memory used                               |
| Status.JVM.Memory.Metaspace.Max                       | Bytes       | Maximum capacity of metaspace                      |
| Status.JVM.GarbageCollector.G1_Old_Generation.Count   | Count       | Count of old generation garbage collections        |
| Status.JVM.Memory.Direct.MemoryUsed                   | Bytes       | Direct memory used                                 |
| Status.JVM.Memory.Mapped.MemoryUsed                   | Bytes       | Mapped memory used                                 |
| Status.JVM.GarbageCollector.G1_Young_Generation.Count | Count       | Count of young generation garbage collections      |
| Status.JVM.Memory.Direct.TotalCapacity                | Bytes       | Total capacity of direct memory                    |
| Status.JVM.GarbageCollector.G1_Old_Generation.Time    | ms          | Time spent on old generation garbage collections   |
| Status.JVM.Memory.Heap.Committed                      | Bytes       | Heap memory committed                              |
| Status.JVM.Memory.Mapped.Count                        | Count       | Count of mapped memory                             |
| Status.JVM.Memory.Metaspace.Used                      | Bytes       | Metaspace memory used                              |
| Status.JVM.Memory.Direct.Count                        | Count       | Count of direct memory                             |
| Status.JVM.Memory.Heap.Used                           | Bytes       | Heap memory used                                   |
| Status.JVM.Memory.Heap.Max                            | Bytes       | Maximum capacity of heap memory                    |
| Status.JVM.GarbageCollector.G1_Young_Generation.Time  | ms          | Time spent on young generation garbage collections |
| Status.JVM.Memory.NonHeap.Max                         | Bytes       | Maximum capacity of non-heap memory                |

#### Metrics Set: JobManager Config

|                Metric Name                 | Metric Unit |              Metric Help Description               |
|--------------------------------------------|-------------|----------------------------------------------------|
| internal.jobgraph-path                     | -           | Internal job graph path                            |
| env.java.home                              | -           | Java environment path                              |
| classloader.check-leaked-classloader       | -           | Whether to check for leaked class loaders          |
| env.java.opts                              | -           | Java options                                       |
| high-availability.cluster-id               | -           | High availability cluster ID                       |
| jobmanager.rpc.address                     | -           | JobManager's RPC address                           |
| jobmanager.memory.jvm-overhead.min         | Bytes       | Minimum JVM overhead for JobManager                |
| jobmanager.web.port                        | Port        | JobManager's Web port                              |
| webclient.port                             | Port        | Web client port                                    |
| execution.savepoint.ignore-unclaimed-state | -           | Whether to ignore unclaimed state                  |
| io.tmp.dirs                                | Path        | Temporary file directories                         |
| parallelism.default                        | -           | Default parallelism                                |
| taskmanager.memory.fraction                | -           | TaskManager memory fraction                        |
| taskmanager.numberOfTaskSlots              | -           | Number of task slots for TaskManager               |
| yarn.application.name                      | -           | Yarn application name                              |
| taskmanager.heap.mb                        | MB          | Heap memory size for TaskManager                   |
| taskmanager.memory.process.size            | GB          | Process memory size for TaskManager                |
| web.port                                   | Port        | Web port                                           |
| classloader.resolve-order                  | -           | Class loader resolve order                         |
| jobmanager.heap.mb                         | MB          | Heap memory size for JobManager                    |
| jobmanager.memory.off-heap.size            | Bytes       | Off-heap memory size for JobManager                |
| state.backend.incremental                  | -           | Whether the state backend is incremental           |
| execution.target                           | -           | Execution target                                   |
| jobmanager.memory.process.size             | GB          | Process memory size for JobManager                 |
| web.tmpdir                                 | Path        | Web temporary directory                            |
| yarn.ship-files                            | Path        | Yarn shipped files                                 |
| jobmanager.rpc.port                        | Port        | JobManager's RPC port                              |
| internal.io.tmpdirs.use-local-default      | -           | Whether to use local default temporary directories |
| execution.checkpointing.interval           | ms          | Checkpointing interval                             |
| execution.attached                         | -           | Whether to execute attached                        |
| internal.cluster.execution-mode            | -           | Internal cluster execution mode                    |
| execution.shutdown-on-attached-exit        | -           | Whether to shutdown on attached exit               |
| pipeline.jars                              | Path        | Pipeline JAR files                                 |
| rest.address                               | -           | REST address                                       |
| state.backend                              | -           | State backend type                                 |
| jobmanager.memory.jvm-metaspace.size       | Bytes       | JVM metaspace size for JobManager                  |
| $internal.deployment.config-dir            | Path        | Internal deployment configuration directory        |
| $internal.yarn.log-config-file             | Path        | Internal Yarn log configuration file path          |
| jobmanager.memory.heap.size                | Bytes       | Heap memory size for JobManager                    |
| state.checkpoints.dir                      | Path        | State checkpoints directory                        |
| jobmanager.memory.jvm-overhead.max         | Bytes       | Maximum JVM overhead for JobManager                |

#### TaskManager Metrics

|              Metric Name              | Metric Unit |              Metric Help Description              |
|---------------------------------------|-------------|---------------------------------------------------|
| Container ID                          | -           | Container ID for uniquely identifying a container |
| Path                                  | -           | Container path                                    |
| Data Port                             | Port        | Data transmission port                            |
| JMX Port                              | Port        | JMX (Java Management Extensions) port             |
| Last Heartbeat                        | Timestamp   | Last heartbeat time                               |
| All Slots                             | Count       | Total number of task slots in the container       |
| Free Slots                            | Count       | Number of free task slots in the container        |
| totalResourceCpuCores                 | Cores       | Total number of CPU cores in the container        |
| totalResourceTaskHeapMemory           | MB          | Total task heap memory size in the container      |
| totalResourceManagedMemory            | MB          | Total managed memory size in the container        |
| totalResourceNetworkMemory            | MB          | Total network memory size in the container        |
| freeResourceCpuCores                  | Cores       | Number of free CPU cores in the container         |
| freeResourceTaskHeapMemory            | MB          | Free task heap memory size in the container       |
| freeResourceTaskOffHeapMemory         | MB          | Free task off-heap memory size in the container   |
| freeResourceManagedMemory             | MB          | Free managed memory size in the container         |
| freeResourceNetworkMemory             | MB          | Free network memory size in the container         |
| CPU Cores                             | Cores       | Number of CPU cores                               |
| Physical MEM                          | MB          | Size of physical memory                           |
| JVM Heap Size                         | MB          | Size of JVM heap memory                           |
| Flink Managed MEM                     | MB          | Size of Flink managed memory                      |
| Framework Heap                        | MB          | Size of framework heap memory                     |
| Task Heap                             | MB          | Size of task heap memory                          |
| Framework Off-Heap                    | MB          | Size of framework off-heap memory                 |
| memoryConfigurationTaskOffHeap        | Bytes       | Task off-heap memory configuration                |
| Network                               | MB          | Network memory configuration                      |
| Managed Memory                        | MB          | Managed memory configuration                      |
| JVM Metaspace                         | MB          | Size of JVM metaspace                             |
| JVM Overhead                          | MB          | JVM overhead                                      |
| memoryConfigurationTotalFlinkMemory   | Bytes       | Total Flink memory configuration                  |
| memoryConfigurationTotalProcessMemory | Bytes       | Total process memory configuration                |

#### TaskManager Status Metrics

|            Metric Name            | Metric Unit |      Metric Help Description       |
|-----------------------------------|-------------|------------------------------------|
| Status.Shuffle.Netty.TotalMemory  | MB          | Total memory used by Netty Shuffle |
| Status.Flink.Memory.Managed.Used  | MB          | Managed memory used by Flink       |
| Status.JVM.Memory.Metaspace.Used  | MB          | Used JVM metaspace memory          |
| Status.JVM.Memory.Metaspace.Max   | MB          | Maximum JVM metaspace memory       |
| Status.JVM.Memory.Heap.Used       | MB          | Used JVM heap memory               |
| Status.JVM.Memory.Heap.Max        | MB          | Maximum JVM heap memory            |
| Status.Flink.Memory.Managed.Total | MB          | Total managed memory by Flink      |
| Status.Shuffle.Netty.UsedMemory   | MB          | Used memory by Netty Shuffle       |