dpdkstat: enable a plugin for DPDK stats
authorMaryam Tahhan <maryam.tahhan@intel.com>
Mon, 28 Mar 2016 11:26:03 +0000 (12:26 +0100)
committerKim Jones <kim-marie.jones@intel.com>
Thu, 28 Jul 2016 11:06:53 +0000 (12:06 +0100)
This patch enables support to retrieve statistics for DPDK
interfaces. An overview of the threading is as follows:
1. collectd init() or read() calls dpdk_helper_spawn() to retrieve
   the required size of xstats to allocate.
   1.1. DPDK counts ports, stats and length, writing them to
        shared-memory (SHM).
   1.2. DPDK helper (secondary) process quits, allowing cleanup of
        the shared memory.
2. collectd resizes shared-memory to size of stats as provided by
   DPDK.
3. collectd respawns the DPDK helper.
   3.1. Helper blocks on a semaphore until told to read the stats
        from DPDK and write them to SHM.
4. collectd dispatches statistics
   4.1. Thread blocks on semaphore until stats are available.
This threading model is required to allow the plugin to detect when
the DPDK primary process/application has been killed/reset, and to avoid
the plugin from stopping another DPDK primary process from starting.
Some extra housekeeping is in place to ensure collectd is never
stalled by using sem_timedwait() with a timeout. If collectd dies, the
helper process will automatically quit after a timeout, as it detects
its ppid has changed.

Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>
Signed-off-by: Maryam Tahhan <maryam.tahhan@intel.com>
README
configure.ac
src/Makefile.am
src/collectd.conf.in
src/collectd.conf.pod
src/dpdkstat.c [new file with mode: 0644]
src/types.db

diff --git a/README b/README
index 1c3af7a..c683df0 100644 (file)
--- a/README
+++ b/README
@@ -988,6 +988,97 @@ Crosscompiling
     * `endianflip' (12345678 -> 87654321)
     * `intswap'    (12345678 -> 56781234)
 
+Configuring with DPDK
+---------------------
+
+Build DPDK for use with collectd:
+  To compile DPDK for use with collectd dpdkstat start by:
+    - Clone DPDK:
+      $ git clone git://dpdk.org/dpdk
+    - Checkout the system requirements at
+      http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html and make sure you have
+      the required tools and hugepage setup as specified there.
+      NOTE: It's recommended to use the 1GB hugepage setup for best performance,
+            please follow the instruction for "Reserving Hugepages for DPDK Use"
+            in the link above.
+      However if you plan on configuring 2MB hugepages on the fly please ensure
+      to add appropriate commands to reserve hugepages in a system startup script
+      if collectd is booted at system startup time. These commands include:
+          mkdir -p /mnt/huge
+          mount -t hugetlbfs nodev /mnt/huge
+          echo 64 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
+    - To configure the DPDK build for the combined shared library modify
+      config/common_base in your DPDK as follows
+        #
+        # Compile to share library
+        #
+        -CONFIG_RTE_BUILD_SHARED_LIB=n
+        +CONFIG_RTE_BUILD_SHARED_LIB=y
+    - Prepare the configuration for the appropriate target as specified at:
+      http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html.
+      For example:
+          $ make config T=x86_64-native-linuxapp-gcc
+    - Build the target:
+          $ make
+    - Install DPDK to /usr
+          $ sudo make install prefix=/usr
+      NOTE 1: You must run make install as the configuration of collectd with
+            DPDK expects DPDK to be installed somewhere.
+      NOTE 2: If you don't specify a prefix then DPDK will be installed in /usr/local/
+      NOTE 3: If you are not root then use sudo to make install DPDK to the appropriate
+            location.
+    - Check that the DPDK library has been installed in /usr/lib or /lib
+          $ ls /usr/lib | grep dpdk
+    - Bind the interfaces to use with dpdkstat to DPDK:
+      DPDK devices can be setup with either the VFIO (for DPDK 1.7+) or UIO modules.
+      Note: UIO requires inserting an out of tree driver igb_uio.ko that is available
+      in DPDK.
+      UIO Setup:
+      - Insert uio.ko:
+           $ sudo modprobe uio
+      - Insert igb_uio.ko:
+           $ sudo insmod $DPDK_BUILD/kmod/igb_uio.ko
+      - Bind network device to igb_uio:
+           $ sudo $DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio eth1
+      VFIO Setup:
+      - VFIO needs to be supported in the kernel and the BIOS. More information can be found
+        @ http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html.
+      - Insert the `vfio-pci.ko' module: modprobe vfio-pci
+      - Set the correct permissions for the vfio device:
+           $ sudo /usr/bin/chmod a+x /dev/vfio
+           $ sudo /usr/bin/chmod 0666 /dev/vfio/*
+     - Bind the network device to vfio-pci:
+           $ sudo $DPDK_DIR/tools/dpdk_nic_bind.py --bind=vfio-pci eth1
+       NOTE: Please ensure to add appropriate commands to bind the network
+             interfaces to DPDK in a system startup script if collectd is
+             booted at system startup time.
+     - Run ldconfig to update the shared library cache.
+
+Configure collectd with DPDK:
+    - Generate the build script as specified below. (i.e. run `build.sh').
+    - Configure collectd with the DPDK library:
+       ./configure --with-libdpdk=/usr
+
+       Libraries:
+       ...
+       dpdk  . . . . . . . . yes
+
+       Modules:
+       ...
+       dpdkstat . . . . . . .yes
+
+
+    - Make sure that dpdk and dpdkstat are enabled in the configuration log:
+
+    - Build collectd:
+          $ make -j && make -j install.
+      NOTE: If you are building on Ubuntu 14.04 you need to use:
+          $  make -j CFLAGS+='-mavx' && make -j install
+
+Usage of dpdkstat:
+    - The same PCI device configuration should be passed to the primary process
+      as the secondary process uses the same port indexes as the primary.
+      NOTE: A blacklist/whitelist of NICs isn't supported yet.
 
 Contact
 -------
index 4aa3af7..c661c56 100644 (file)
@@ -2411,6 +2411,69 @@ then
 fi
 # }}}
 
+# --with-libdpdk {{{
+AC_ARG_WITH(libdpdk, [AS_HELP_STRING([--with-libdpdk@<:@=PREFIX@:>@], [Path to the DPDK build directory.])],
+[
+  if test "x$withval" != "xno" && test "x$withval" != "xyes"
+       then
+               with_dpdk_path="$withval"
+               with_dpdk="yes"
+       else
+               if test "x$withval" = "xno"
+               then
+                       with_dpdk="no (disabled)"
+               fi
+       fi
+], [with_dpdk="yes"])
+
+if test "x$with_dpdk" = "xyes"
+then
+       RTE_BUILD="$with_dpdk_path"
+       DPDK_INCLUDE="$RTE_BUILD/include/dpdk"
+       DPDK_LIB_DIR="$RTE_BUILD/lib"
+       DPDK_LIB="-ldpdk"
+       DPDK_EXTRA_LIB="-ldl -lpthread -lrt -lm"
+
+       SAVE_CFLAGS="$CFLAGS"
+       SAVE_LDFLAGS="$LDFLAGS"
+       LDFLAGS="$LDFLAGS -L$DPDK_LIB_DIR"
+       CFLAGS="$CFLAGS -I$DPDK_INCLUDE -fpic"
+       FOUND_DPDK=false
+       SAVE_LIBS="$LIBS"
+       LIBS="$LIBS $DPDK_LIB $DPDK_EXTRA_LIB"
+       AC_CHECK_LIB(dpdk, rte_eal_init,
+                     [with_libdpdk="yes"],
+                     [with_libdpdk="DPDK not found"])
+       AC_LINK_IFELSE(
+               [AC_LANG_PROGRAM(
+[[
+#include <rte_config.h>
+#include <rte_eal.h>
+]],
+[[
+int rte_argc;
+char ** rte_argv;
+rte_eal_init(rte_argc, rte_argv);
+]]
+       )],
+               [FOUND_DPDK=true])
+       if $FOUND_DPDK; then :; else
+                AC_MSG_ERROR([cannot link with dpdk])
+       fi
+       DPDK_LDFLAGS="-ldpdk"
+       CFLAGS="$SAVE_CFLAGS"
+       LDFLAGS="$SAVE_LDFLAGS"
+       LIBS="$SAVE_LIBS"
+       BUILD_WITH_DPDK_CFLAGS="-I$DPDK_INCLUDE -fPIC"
+       BUILD_WITH_DPDK_LDFLAGS="-L$DPDK_LIB_DIR"
+       BUILD_WITH_DPDK_LIBS="$DPDK_LDFLAGS -ldl -lpthread -lrt -lm"
+       AC_SUBST(BUILD_WITH_DPDK_CFLAGS)
+       AC_SUBST(BUILD_WITH_DPDK_LDFLAGS)
+       AC_SUBST(BUILD_WITH_DPDK_LIBS)
+       AC_DEFINE_UNQUOTED(DPDK, [1], [System uses DPDK.])
+fi
+# }}}
+
 # --with-java {{{
 with_java_home="$JAVA_HOME"
 if test "x$with_java_home" = "x"
@@ -5556,6 +5619,7 @@ plugin_curl_xml="no"
 plugin_df="no"
 plugin_disk="no"
 plugin_drbd="no"
+plugin_dpdk="no"
 plugin_entropy="no"
 plugin_ethstat="no"
 plugin_fhcount="no"
@@ -5974,6 +6038,7 @@ AC_PLUGIN([dbi],                 [$with_libdbi],            [General database st
 AC_PLUGIN([df],                  [$plugin_df],              [Filesystem usage statistics])
 AC_PLUGIN([disk],                [$plugin_disk],            [Disk usage statistics])
 AC_PLUGIN([dns],                 [$with_libpcap],           [DNS traffic analysis])
+AC_PLUGIN([dpdkstat],    [$with_libdpdk],      [Stats & Status from DPDK])
 AC_PLUGIN([drbd],                [$plugin_drbd],            [DRBD statistics])
 AC_PLUGIN([email],               [yes],                     [EMail statistics])
 AC_PLUGIN([entropy],             [$plugin_entropy],         [Entropy statistics])
@@ -6293,6 +6358,7 @@ AC_MSG_RESULT([    PROTOC  . . . . . . . $PROTOC])
 AC_MSG_RESULT([    YACC  . . . . . . . . $YACC])
 AC_MSG_RESULT([    YFLAGS  . . . . . . . $YFLAGS])
 AC_MSG_RESULT()
+    dpdk  . . . . . . . . $with_libdpdk
 AC_MSG_RESULT([  Libraries:])
 AC_MSG_RESULT([    intel mic . . . . . . $with_mic])
 AC_MSG_RESULT([    libaquaero5 . . . . . $with_libaquaero5])
@@ -6382,6 +6448,7 @@ AC_MSG_RESULT([    curl  . . . . . . . . $enable_curl])
 AC_MSG_RESULT([    curl_json . . . . . . $enable_curl_json])
 AC_MSG_RESULT([    curl_xml  . . . . . . $enable_curl_xml])
 AC_MSG_RESULT([    dbi . . . . . . . . . $enable_dbi])
+    dpdkstat . . . . . . .$enable_dpdkstat
 AC_MSG_RESULT([    df  . . . . . . . . . $enable_df])
 AC_MSG_RESULT([    disk  . . . . . . . . $enable_disk])
 AC_MSG_RESULT([    dns . . . . . . . . . $enable_dns])
index 93045e2..8d9016f 100644 (file)
@@ -369,6 +369,14 @@ dns_la_LDFLAGS = $(PLUGIN_LDFLAGS)
 dns_la_LIBADD = -lpcap
 endif
 
+if BUILD_PLUGIN_DPDKSTAT
+pkglib_LTLIBRARIES += dpdkstat.la
+dpdkstat_la_SOURCES = dpdkstat.c
+dpdkstat_la_CFLAGS = $(AM_CFLAGS) $(BUILD_WITH_DPDK_CFLAGS)
+dpdkstat_la_LDFLAGS = $(PLUGIN_LDFLAGS) $(BUILD_WITH_DPDK_LDFLAGS) $(BUILD_WITH_DPDK_LIBS)
+dpdkstat_la_LIBADD = -ldl -lpthread -lrt -lm
+endif
+
 if BUILD_PLUGIN_DRBD
 pkglib_LTLIBRARIES += drbd.la
 drbd_la_SOURCES = drbd.c
index 54072d0..790b242 100644 (file)
 #@BUILD_PLUGIN_DF_TRUE@LoadPlugin df
 #@BUILD_PLUGIN_DISK_TRUE@LoadPlugin disk
 #@BUILD_PLUGIN_DNS_TRUE@LoadPlugin dns
+#@BUILD_PLUGIN_DPDKSTAT_TRUE@LoadPlugin dpdkstat
 #@BUILD_PLUGIN_DRBD_TRUE@LoadPlugin drbd
 #@BUILD_PLUGIN_EMAIL_TRUE@LoadPlugin email
 #@BUILD_PLUGIN_ENTROPY_TRUE@LoadPlugin entropy
 #      SelectNumericQueryTypes true
 #</Plugin>
 
+#<Plugin dpdkstat>
+#       Interval 1
+#       Coremask "0xf"
+#       ProcessType "secondary"
+#       FilePrefix "rte"
+#</Plugin>
+
 #<Plugin email>
 #      SocketFile "@localstatedir@/run/@PACKAGE_NAME@-email"
 #      SocketGroup "collectd"
index 04c654a..6aabba6 100644 (file)
@@ -2343,6 +2343,51 @@ Enabled by default, collects unknown (and thus presented as numeric only) query
 
 =back
 
+=head2 Plugin C<dpdkstat>
+
+The I<dpdkstat plugin> collects information about DPDK interfaces using the
+extended NIC stats API in DPDK.
+
+B<Synopsis:>
+
+ <Plugin "dpdkstat">
+    Interval 1
+    Coremask "0x4"
+    MemoryChannels "4"
+    ProcessType "secondary"
+    FilePrefix "rte"
+ </Plugin>
+
+B<Options:>
+
+=over 4
+
+=item B<Interval> I<seconds>
+
+The interval within which to retreive stats in seconds. For milliseconds simple
+divide the time by 1000 for example if the desired interval is 50ms, set
+interval to 0.05.
+
+=item B<Coremask> I<Mask>
+
+An hexadecimal bit mask of the cores to run on. Note that core numbering can
+change between platforms and should be determined beforehand.
+
+=item B<Memorychannels> I<Channels>
+
+Number of memory channels per processor socket.
+
+=item B<ProcessType> I<type>
+
+The type of DPDK process instance.
+
+=item B<FilePrefix> I<File>
+
+The prefix text used for hugepage filenames. The filename will be set to
+/var/run/.<prefix>_config where prefix is what is passed in by the user.
+
+=back
+
 =head2 Plugin C<email>
 
 =over 4
diff --git a/src/dpdkstat.c b/src/dpdkstat.c
new file mode 100644 (file)
index 0000000..27e2d39
--- /dev/null
@@ -0,0 +1,670 @@
+/*-
+ * collectd - src/dpdkstat.c
+ * MIT License
+ *
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy of
+ * this software and associated documentation files (the "Software"), to deal in
+ * the Software without restriction, including without limitation the rights to
+ * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+ * of the Software, and to permit persons to whom the Software is furnished to do
+ * so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in all
+ * copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ * Authors:
+ *   Maryam Tahhan <maryam.tahhan@intel.com>
+ *   Harry van Haaren <harry.van.haaren@intel.com>
+ */
+
+#include "collectd.h"
+#include "common.h" /* auxiliary functions */
+#include "plugin.h" /* plugin_register_*, plugin_dispatch_values */
+#include "utils_time.h"
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <signal.h>
+#include <stdarg.h>
+#include <stdlib.h>
+#include <errno.h>
+#include <getopt.h>
+#include <inttypes.h>
+#include <fcntl.h>
+#include <semaphore.h>
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <sys/time.h>
+#include <sys/wait.h>
+#include <poll.h>
+#include <unistd.h>
+#include <string.h>
+
+#include <rte_config.h>
+#include <rte_eal.h>
+#include <rte_ethdev.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_malloc.h>
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_tailq.h>
+#include <rte_lcore.h>
+#include <rte_per_lcore.h>
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_atomic.h>
+#include <rte_branch_prediction.h>
+#include <rte_string_fns.h>
+
+
+#define DATA_MAX_NAME_LEN        64
+#define DPDKSTAT_MAX_BUFFER_SIZE (4096*4)
+#define DPDK_SHM_NAME "dpdk_collectd_stats_shm"
+#define REINIT_SHM 1
+#define RESET 1
+#define NO_RESET 0
+
+enum DPDK_HELPER_ACTION {
+  DPDK_HELPER_ACTION_COUNT_STATS,
+  DPDK_HELPER_ACTION_SEND_STATS,
+};
+
+enum DPDK_HELPER_STATUS {
+  DPDK_HELPER_NOT_INITIALIZED = 0,
+  DPDK_HELPER_WAITING_ON_PRIMARY,
+  DPDK_HELPER_INITIALIZING_EAL,
+  DPDK_HELPER_ALIVE_SENDING_STATS,
+  DPDK_HELPER_GRACEFUL_QUIT,
+};
+
+struct dpdk_config_s {
+  /* General DPDK params */
+  char coremask[DATA_MAX_NAME_LEN];
+  char memory_channels[DATA_MAX_NAME_LEN];
+  char socket_memory[DATA_MAX_NAME_LEN];
+  char process_type[DATA_MAX_NAME_LEN];
+  char file_prefix[DATA_MAX_NAME_LEN];
+  cdtime_t interval;
+  uint32_t eal_initialized;
+  uint32_t enabled_port_mask;
+  uint32_t eal_argc;
+  /* Helper info */
+  int   collectd_reinit_shm;
+  pid_t helper_pid;
+  sem_t sema_helper_get_stats;
+  sem_t sema_stats_in_shm;
+  int   helper_pipes[2];
+  enum DPDK_HELPER_STATUS helper_status;
+  enum DPDK_HELPER_ACTION helper_action;
+  /* xstats info */
+  uint32_t num_ports;
+  uint32_t num_xstats;
+  cdtime_t port_read_time[RTE_MAX_ETHPORTS];
+  uint32_t num_stats_in_port[RTE_MAX_ETHPORTS];
+  struct rte_eth_link link_status[RTE_MAX_ETHPORTS];
+  struct rte_eth_xstats xstats;
+  /* rte_eth_xstats from here on until the end of the SHM */
+};
+typedef struct dpdk_config_s dpdk_config_t;
+
+static int g_configured = 0;
+static dpdk_config_t *g_configuration = 0;
+
+static int dpdk_config_init_default(void);
+static int dpdk_config(oconfig_item_t *ci);
+static int dpdk_helper_init_eal(void);
+static int dpdk_helper_run(void);
+static int dpdk_helper_spawn(enum DPDK_HELPER_ACTION action);
+static int dpdk_init (void);
+static int dpdk_read(user_data_t *ud);
+static int dpdk_shm_cleanup(void);
+static int dpdk_shm_init(size_t size);
+void module_register(void);
+
+/* Write the default configuration to the g_configuration instances */
+static int dpdk_config_init_default(void)
+{
+    g_configuration->interval = plugin_get_interval();
+    WARNING("dpdkstat: No time interval was configured, default value %lu ms is set\n",
+             CDTIME_T_TO_MS(g_configuration->interval));
+    g_configuration->enabled_port_mask = 0;
+    g_configuration->eal_argc = 2;
+    g_configuration->eal_initialized = 0;
+    snprintf(g_configuration->coremask, DATA_MAX_NAME_LEN, "%s", "0xf");
+    snprintf(g_configuration->memory_channels, DATA_MAX_NAME_LEN, "%s", "1");
+    snprintf(g_configuration->process_type, DATA_MAX_NAME_LEN, "%s", "secondary");
+    snprintf(g_configuration->file_prefix, DATA_MAX_NAME_LEN, "%s",
+             "/var/run/.rte_config");
+  return 0;
+}
+
+static int dpdk_config(oconfig_item_t *ci)
+{
+  int i = 0, ret = 0;
+
+  /* Initialize a POSIX SHared Memory (SHM) object. */
+  dpdk_shm_init(sizeof(dpdk_config_t));
+
+  /* Set defaults for config, overwritten by loop if config item exists */
+  ret = dpdk_config_init_default();
+  if(ret != 0) {
+    return -1;
+  }
+
+  for (i = 0; i < ci->children_num; i++) {
+    oconfig_item_t *child = ci->children + i;
+
+    if (strcasecmp("Interval", child->key) == 0) {
+      g_configuration->interval =
+            DOUBLE_TO_CDTIME_T (child->values[0].value.number);
+      DEBUG("dpdkstat: Plugin Read Interval %lu milliseconds\n",
+            CDTIME_T_TO_MS(g_configuration->interval));
+    } else if (strcasecmp("Coremask", child->key) == 0) {
+      snprintf(g_configuration->coremask, DATA_MAX_NAME_LEN, "%s",
+               child->values[0].value.string);
+      DEBUG("dpdkstat:COREMASK %s \n", g_configuration->coremask);
+      g_configuration->eal_argc+=1;
+    } else if (strcasecmp("MemoryChannels", child->key) == 0) {
+      snprintf(g_configuration->memory_channels, DATA_MAX_NAME_LEN, "%s",
+               child->values[0].value.string);
+      DEBUG("dpdkstat:Memory Channels %s \n", g_configuration->memory_channels);
+      g_configuration->eal_argc+=1;
+    } else if (strcasecmp("SocketMemory", child->key) == 0) {
+      snprintf(g_configuration->socket_memory, DATA_MAX_NAME_LEN, "%s",
+               child->values[0].value.string);
+      DEBUG("dpdkstat: socket mem %s \n", g_configuration->socket_memory);
+      g_configuration->eal_argc+=1;
+    } else if (strcasecmp("ProcessType", child->key) == 0) {
+      snprintf(g_configuration->process_type, DATA_MAX_NAME_LEN, "%s",
+               child->values[0].value.string);
+      DEBUG("dpdkstat: proc type %s \n", g_configuration->process_type);
+      g_configuration->eal_argc+=1;
+    } else if (strcasecmp("FilePrefix", child->key) == 0) {
+      snprintf(g_configuration->file_prefix, DATA_MAX_NAME_LEN, "/var/run/.%s_config",
+               child->values[0].value.string);
+      DEBUG("dpdkstat: file prefix %s \n", g_configuration->file_prefix);
+      if (strcasecmp(g_configuration->file_prefix, "/var/run/.rte_config") != 0) {
+        g_configuration->eal_argc+=1;
+      }
+    } else {
+      WARNING ("dpdkstat: The config option \"%s\" is unknown.",
+               child->key);
+    }
+  } /* End for (i = 0; i < ci->children_num; i++)*/
+  g_configured = 1; /* Bypass configuration in dpdk_shm_init(). */
+
+  return 0;
+}
+
+/* Initialize SHared Memory (SHM) for config and helper process */
+static int dpdk_shm_init(size_t size)
+{
+  /*
+   * Check if SHM is already configured: when config items are provided, the
+   * config function initializes SHM. If there is no config, then init() will
+   * just return.
+   */
+  if(g_configuration)
+    return 0;
+
+  /* Create and open a new object, or open an existing object. */
+  int fd = shm_open(DPDK_SHM_NAME, O_CREAT | O_TRUNC | O_RDWR, 0666);
+  if (fd < 0) {
+    WARNING("dpdkstat:Failed to open %s as SHM:%s\n", DPDK_SHM_NAME,
+            strerror(errno));
+    goto fail;
+  }
+  /* Set the size of the shared memory object. */
+  int ret = ftruncate(fd, size);
+  if (ret != 0) {
+    WARNING("dpdkstat:Failed to resize SHM:%s\n", strerror(errno));
+    goto fail_close;
+  }
+  /* Map the shared memory object into this process' virtual address space. */
+  g_configuration = (dpdk_config_t *)
+                    mmap(0, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+  if (g_configuration == MAP_FAILED) {
+    WARNING("dpdkstat:Failed to mmap SHM:%s\n", strerror(errno));
+    goto fail_close;
+  }
+  /*
+   * Close the file descriptor, the shared memory object still exists
+   * and can only be removed by calling shm_unlink().
+   */
+  close(fd);
+
+  /* Initialize g_configuration. */
+  memset(g_configuration, 0, size);
+
+  /* Initialize the semaphores for SHM use */
+  sem_init(&g_configuration->sema_helper_get_stats, 1, 0);
+  sem_init(&g_configuration->sema_stats_in_shm, 1, 0);
+  return 0;
+
+fail_close:
+  close(fd);
+fail:
+  /* Reset to zero, as it was set to MAP_FAILED aka: (void *)-1. Avoid
+   * an issue if collectd attempts to run this plugin failure.
+   */
+  g_configuration = 0;
+  return -1;
+}
+
+static int dpdk_re_init_shm()
+{
+  dpdk_config_t temp_config;
+  memcpy(&temp_config,g_configuration, sizeof(dpdk_config_t));
+  DEBUG("dpdkstat: %s: ports %d, xstats %d\n", __func__, temp_config.num_ports,
+        temp_config.num_xstats);
+
+  int shm_xstats_size = sizeof(dpdk_config_t) + (sizeof(struct rte_eth_xstats) *
+                        g_configuration->num_xstats);
+  DEBUG("=== SHM new size for %d xstats\n", g_configuration->num_xstats);
+
+  int err = dpdk_shm_cleanup();
+  if (err)
+    ERROR("dpdkstat: Error in shm_cleanup in %s\n", __func__);
+
+  err = dpdk_shm_init(shm_xstats_size);
+  if (err)
+    ERROR("dpdkstat: Error in shm_init in %s\n", __func__);
+
+  /* If the XML config() function has been run, dont re-initialize defaults */
+  if(!g_configured)
+    dpdk_config_init_default();
+
+  memcpy(g_configuration,&temp_config, sizeof(dpdk_config_t));
+  g_configuration->collectd_reinit_shm = 0;
+
+  return 0;
+}
+
+static int dpdk_init (void)
+{
+  int ret = 0;
+  int err = dpdk_shm_init(sizeof(dpdk_config_t));
+  if (err)
+    ERROR("dpdkstat: %s : error %d in shm_init()", __func__, err);
+
+  /* If the XML config() function has been run, dont re-initialize defaults */
+  if(!g_configured) {
+    ret = dpdk_config_init_default();
+    if (ret != 0) {
+      return -1;
+    }
+  }
+
+  plugin_register_complex_read (NULL, "dpdkstat", dpdk_read,
+                                g_configuration->interval, NULL);
+  return 0;
+}
+
+static int dpdk_helper_exit(int reset)
+{
+  g_configuration->helper_status = DPDK_HELPER_GRACEFUL_QUIT;
+  if(reset) {
+    g_configuration->eal_initialized = 0;
+    g_configuration->num_ports = 0;
+    memset(&g_configuration->xstats, 0, g_configuration->num_xstats* sizeof(struct rte_eth_xstats));
+    g_configuration->num_xstats = 0;
+    int i =0;
+    for(;i < RTE_MAX_ETHPORTS; i++)
+      g_configuration->num_stats_in_port[i] = 0;
+  }
+  close(g_configuration->helper_pipes[1]);
+  kill(g_configuration->helper_pid, SIGKILL);
+  return 0;
+}
+
+static int dpdk_helper_spawn(enum DPDK_HELPER_ACTION action)
+{
+  g_configuration->eal_initialized = 0;
+  g_configuration->helper_action = action;
+  /*
+   * Create a pipe for helper stdout back to collectd. This is necessary for
+   * logging EAL failures, as rte_eal_init() calls rte_panic().
+   */
+  if(g_configuration->helper_pipes[1]) {
+    DEBUG("dpdkstat: collectd closing helper pipe %d\n",
+          g_configuration->helper_pipes[1]);
+  } else {
+    DEBUG("dpdkstat: collectd helper pipe %d, not closing\n",
+          g_configuration->helper_pipes[1]);
+  }
+  if(pipe(g_configuration->helper_pipes) != 0) {
+    DEBUG("dpdkstat: Could not create helper pipe: %s\n", strerror(errno));
+    return -1;
+  }
+
+  int pipe0_flags = fcntl(g_configuration->helper_pipes[1], F_GETFL, 0);
+  int pipe1_flags = fcntl(g_configuration->helper_pipes[0], F_GETFL, 0);
+  fcntl(g_configuration->helper_pipes[1], F_SETFL, pipe1_flags | O_NONBLOCK);
+  fcntl(g_configuration->helper_pipes[0], F_SETFL, pipe0_flags | O_NONBLOCK);
+
+  pid_t pid = fork();
+  if (pid > 0) {
+    close(g_configuration->helper_pipes[1]);
+    g_configuration->helper_pid = pid;
+    DEBUG("dpdkstat: helper pid %u\n", g_configuration->helper_pid);
+    /* Kick helper once its alive to have it start processing */
+    sem_post(&g_configuration->sema_helper_get_stats);
+  } else if (pid == 0) {
+    /* Replace stdout with a pipe to collectd. */
+    close(g_configuration->helper_pipes[0]);
+    close(STDOUT_FILENO);
+    dup2(g_configuration->helper_pipes[1], STDOUT_FILENO);
+    dpdk_helper_run();
+  } else {
+    ERROR("dpdkstat: Failed to fork helper process: %s\n", strerror(errno));
+    return -1;
+  }
+  return 0;
+}
+
+/*
+ * Initialize the DPDK EAL, if this returns, EAL is successfully initialized.
+ * On failure, the EAL prints an error message, and the helper process exits.
+ */
+static int dpdk_helper_init_eal(void)
+{
+  g_configuration->helper_status = DPDK_HELPER_INITIALIZING_EAL;
+  char *argp[(g_configuration->eal_argc) + 1];
+  int i = 0;
+
+  argp[i++] = "collectd-dpdk";
+  if(strcasecmp(g_configuration->coremask, "") != 0) {
+    argp[i++] = "-c";
+    argp[i++] = g_configuration->coremask;
+  }
+  if(strcasecmp(g_configuration->memory_channels, "") != 0) {
+    argp[i++] = "-n";
+    argp[i++] = g_configuration->memory_channels;
+  }
+  if(strcasecmp(g_configuration->socket_memory, "") != 0) {
+    argp[i++] = "--socket-mem";
+    argp[i++] = g_configuration->socket_memory;
+  }
+  if(strcasecmp(g_configuration->file_prefix, "") != 0 &&
+     strcasecmp(g_configuration->file_prefix, "/var/run/.rte_config") != 0) {
+    argp[i++] = "--file-prefix";
+    argp[i++] = g_configuration->file_prefix;
+  }
+  if(strcasecmp(g_configuration->process_type, "") != 0) {
+    argp[i++] = "--proc-type";
+    argp[i++] = g_configuration->process_type;
+  }
+  g_configuration->eal_argc = i;
+
+  g_configuration->eal_initialized = 1;
+  int ret = rte_eal_init(g_configuration->eal_argc, argp);
+  if (ret < 0) {
+    g_configuration->eal_initialized = 0;
+    printf("dpdkstat: ERROR initializing EAL ret = %d\n", ret);
+    printf("dpdkstat: EAL arguments: ");
+    for (i=0; i< g_configuration->eal_argc; i++) {
+      printf("%s ", argp[i]);
+    }
+    printf("\n");
+    return -1;
+  }
+  return 0;
+}
+
+static int dpdk_helper_run (void)
+{
+  pid_t ppid = getppid();
+  g_configuration->helper_status = DPDK_HELPER_WAITING_ON_PRIMARY;
+
+   while(1) {
+    /* sem_timedwait() to avoid blocking forever */
+    struct timespec ts;
+    cdtime_t now = cdtime();
+    cdtime_t half_sec = MS_TO_CDTIME_T(1500);
+    CDTIME_T_TO_TIMESPEC(now + half_sec + g_configuration->interval *2, &ts);
+    int ret = sem_timedwait(&g_configuration->sema_helper_get_stats, &ts);
+
+    if(ret == -1 && errno == ETIMEDOUT) {
+      ERROR("dpdkstat-helper: sem timedwait()"
+             " timeout, did collectd terminate?\n");
+      dpdk_helper_exit(RESET);
+    }
+
+    /* Parent PID change means collectd died so quit the helper process. */
+    if (ppid != getppid()) {
+      WARNING("dpdkstat-helper: parent PID changed, quitting.\n");
+      dpdk_helper_exit(RESET);
+    }
+
+    /* Checking for DPDK primary process. */
+    if (!rte_eal_primary_proc_alive(g_configuration->file_prefix)) {
+      if(g_configuration->eal_initialized) {
+        WARNING("dpdkstat-helper: no primary alive but EAL initialized:"
+              " quitting.\n");
+        dpdk_helper_exit(RESET);
+      }
+      g_configuration->helper_status = DPDK_HELPER_WAITING_ON_PRIMARY;
+      /* Back to start of while() - waiting for primary process */
+      continue;
+    }
+
+    if(!g_configuration->eal_initialized) {
+      /* Initialize EAL. */
+      int ret = dpdk_helper_init_eal();
+      if(ret != 0)
+        dpdk_helper_exit(RESET);
+    }
+
+    g_configuration->helper_status = DPDK_HELPER_ALIVE_SENDING_STATS;
+
+    uint8_t nb_ports;
+    nb_ports = rte_eth_dev_count();
+    if (nb_ports == 0) {
+      DEBUG("dpdkstat-helper: No DPDK ports available. "
+              "Check bound devices to DPDK driver.\n");
+      return 0;
+    }
+
+    if (nb_ports > RTE_MAX_ETHPORTS)
+      nb_ports = RTE_MAX_ETHPORTS;
+    /* If no port mask was specified enable all ports*/
+    if (g_configuration->enabled_port_mask == 0)
+      g_configuration->enabled_port_mask = 0xffff;
+
+    int i, len = 0, enabled_port_count = 0, num_xstats = 0;
+    for (i = 0; i < nb_ports; i++) {
+      if (g_configuration->enabled_port_mask & (1 << i)) {
+        if(g_configuration->helper_action == DPDK_HELPER_ACTION_COUNT_STATS) {
+          len = rte_eth_xstats_get(i, NULL, 0);
+          if (len < 0) {
+            ERROR("dpdkstat-helper: Cannot get xstats count\n");
+            return -1;
+          }
+          num_xstats += len;
+          g_configuration->num_stats_in_port[enabled_port_count] = len;
+          enabled_port_count++;
+          continue;
+        } else {
+          len = g_configuration->num_stats_in_port[enabled_port_count];
+          g_configuration->port_read_time[enabled_port_count] = cdtime();
+          ret = rte_eth_xstats_get(i, &g_configuration->xstats + num_xstats,
+                                   g_configuration->num_stats_in_port[i]);
+          if (ret < 0 || ret != len) {
+            DEBUG("dpdkstat-helper: Error reading xstats on port %d len = %d\n",
+                  i, len);
+            return -1;
+          }
+          num_xstats += g_configuration->num_stats_in_port[i];
+        }
+      } /* if (enabled_port_mask) */
+    } /* for (nb_ports) */
+
+    if(g_configuration->helper_action == DPDK_HELPER_ACTION_COUNT_STATS) {
+      g_configuration->num_ports  = enabled_port_count;
+      g_configuration->num_xstats = num_xstats;
+      DEBUG("dpdkstat-helper ports: %d, num stats: %d\n",
+            g_configuration->num_ports, g_configuration->num_xstats);
+      /* Exit, allowing collectd to re-init SHM to the right size */
+      g_configuration->collectd_reinit_shm = REINIT_SHM;
+      dpdk_helper_exit(NO_RESET);
+    }
+    /* Now kick collectd send thread to send the stats */
+    sem_post(&g_configuration->sema_stats_in_shm);
+  } /* while(1) */
+
+  return 0;
+}
+
+static int dpdk_read (user_data_t *ud)
+{
+  int ret = 0;
+
+  /*
+   * Check if SHM flag is set to be re-initialized. AKA DPDK ports have been
+   * counted, so re-init SHM to be large enough to fit all the statistics.
+   */
+  if(g_configuration->collectd_reinit_shm) {
+    DEBUG("dpdkstat: read() now reinit SHM then launching send-thread\n");
+    dpdk_re_init_shm();
+  }
+
+  /*
+   * Check if DPDK proc is alive, and has already counted port / stats. This
+   * must be done in dpdk_read(), because the DPDK primary process may not be
+   * alive at dpdk_init() time.
+   */
+  if(g_configuration->helper_status == DPDK_HELPER_NOT_INITIALIZED ||
+     g_configuration->helper_status == DPDK_HELPER_GRACEFUL_QUIT) {
+      int action = DPDK_HELPER_ACTION_SEND_STATS;
+      if(g_configuration->num_xstats == 0)
+        action = DPDK_HELPER_ACTION_COUNT_STATS;
+      /* Spawn the helper thread to count stats or to read stats. */
+      dpdk_helper_spawn(action);
+    }
+
+  int exit_status;
+  pid_t ws = waitpid(g_configuration->helper_pid, &exit_status, WNOHANG);
+  /*
+   * Conditions under which to respawn helper:
+   *  waitpid() fails, helper process died (or quit), so respawn
+   */
+  int respawn_helper = 0;
+  if(ws != 0) {
+    respawn_helper = 1;
+  }
+
+  char buf[DPDKSTAT_MAX_BUFFER_SIZE];
+  char out[DPDKSTAT_MAX_BUFFER_SIZE];
+
+  /* non blocking check on helper logging pipe */
+  struct pollfd fds;
+  fds.fd = g_configuration->helper_pipes[0];
+  fds.events = POLLIN;
+  int data_avail = poll(&fds, 1, 0);
+  while(data_avail) {
+    int nbytes = read(g_configuration->helper_pipes[0], buf, sizeof(buf));
+    if(nbytes <= 0)
+      break;
+    snprintf( out, nbytes, "%s", buf);
+    DEBUG("dpdkstat: helper-proc: %s\n", out);
+  }
+
+  if(respawn_helper) {
+    if (g_configuration->helper_pid)
+      dpdk_helper_exit(RESET);
+    dpdk_helper_spawn(DPDK_HELPER_ACTION_COUNT_STATS);
+  }
+
+  struct timespec helper_kick_time;
+  clock_gettime(CLOCK_REALTIME, &helper_kick_time);
+  /* Kick helper process through SHM */
+  sem_post(&g_configuration->sema_helper_get_stats);
+
+  struct timespec ts;
+  cdtime_t now = cdtime();
+  CDTIME_T_TO_TIMESPEC(now + g_configuration->interval, &ts);
+  ret = sem_timedwait(&g_configuration->sema_stats_in_shm, &ts);
+  if(ret == -1 && errno == ETIMEDOUT) {
+    DEBUG("dpdkstat: timeout in collectd thread: is a DPDK Primary running? \n");
+    return 0;
+  }
+
+  /* Dispatch the stats.*/
+    int i, j, count = 0;
+
+    for (i = 0; i < g_configuration->num_ports; i++) {
+      cdtime_t time = g_configuration->port_read_time[i];
+      char dev_name[64];
+      int len = g_configuration->num_stats_in_port[i];
+      snprintf(dev_name, sizeof(dev_name), "port.%d", i);
+      struct rte_eth_xstats *xstats = (&g_configuration->xstats);
+      xstats += count; /* pointer arithmetic to jump to each stats struct */
+      for (j = 0; j < len; j++) {
+        value_t dpdkstat_values[1];
+        value_list_t dpdkstat_vl = VALUE_LIST_INIT;
+
+        dpdkstat_values[0].counter = xstats[j].value;
+        dpdkstat_vl.values = dpdkstat_values;
+        dpdkstat_vl.values_len = 1; /* Submit stats one at a time */
+        dpdkstat_vl.time = time;
+        sstrncpy (dpdkstat_vl.host, hostname_g, sizeof (dpdkstat_vl.host));
+        sstrncpy (dpdkstat_vl.plugin, "dpdkstat", sizeof (dpdkstat_vl.plugin));
+        sstrncpy (dpdkstat_vl.plugin_instance, dev_name,
+                  sizeof (dpdkstat_vl.plugin_instance));
+        sstrncpy (dpdkstat_vl.type, "counter",
+                  sizeof (dpdkstat_vl.type));
+        sstrncpy (dpdkstat_vl.type_instance, xstats[j].name,
+                  sizeof (dpdkstat_vl.type_instance));
+        plugin_dispatch_values (&dpdkstat_vl);
+      }
+      count += len;
+    } /* for each port */
+  return 0;
+}
+
+static int dpdk_shm_cleanup(void)
+{
+  int ret = munmap(g_configuration, sizeof(dpdk_config_t));
+  g_configuration = 0;
+  if(ret) {
+    WARNING("dpdkstat: munmap returned %d\n", ret);
+    return ret;
+  }
+  ret = shm_unlink(DPDK_SHM_NAME);
+  if(ret) {
+    WARNING("dpdkstat: shm_unlink returned %d\n", ret);
+    return ret;
+  }
+  return 0;
+}
+
+static int dpdk_shutdown (void)
+{
+  close(g_configuration->helper_pipes[1]);
+  kill(g_configuration->helper_pid, SIGKILL);
+  int ret = dpdk_shm_cleanup();
+
+  return ret;
+}
+
+void module_register (void)
+{
+  plugin_register_complex_config ("dpdkstat", dpdk_config);
+  plugin_register_init ("dpdkstat", dpdk_init);
+  plugin_register_shutdown ("dpdkstat", dpdk_shutdown);
+}
index 8c6a995..392a586 100644 (file)
@@ -259,3 +259,134 @@ arc_ratio               value:GAUGE:0:U
 arc_size                current:GAUGE:0:U, target:GAUGE:0:U, minlimit:GAUGE:0:U, maxlimit:GAUGE:0:U
 mysql_qcache            hits:COUNTER:0:U, inserts:COUNTER:0:U, not_cached:COUNTER:0:U, lowmem_prunes:COUNTER:0:U, queries_in_cache:GAUGE:0:U
 mysql_threads           running:GAUGE:0:U, connected:GAUGE:0:U, cached:GAUGE:0:U, created:COUNTER:0:U
+
+#
+#DPDK 10GB NIC Types
+#
+
+rx_good_packets value:COUNTER:0:U
+tx_good_packets value:COUNTER:0:U
+rx_good_bytes value:COUNTER:0:U
+tx_good_bytes value:COUNTER:0:U
+rx_errors value:COUNTER:0:U
+tx_errors value:COUNTER:0:U
+rx_mbuf_allocation_errors value:COUNTER:0:U
+rx_q0_packets value:COUNTER:0:U
+rx_q0_bytes value:COUNTER:0:U
+rx_q0_errors value:COUNTER:0:U
+tx_q0_packets value:COUNTER:0:U
+tx_q0_bytes value:COUNTER:0:U
+rx_crc_errors value:COUNTER:0:U
+rx_illegal_byte_errors value:COUNTER:0:U
+rx_error_bytes value:COUNTER:0:U
+mac_local_errors value:COUNTER:0:U
+mac_remote_errors value:COUNTER:0:U
+rx_length_errors value:COUNTER:0:U
+tx_xon_packets value:COUNTER:0:U
+rx_xon_packets value:COUNTER:0:U
+tx_xoff_packets value:COUNTER:0:U
+rx_xoff_packets value:COUNTER:0:U
+rx_size_64_packets value:COUNTER:0:U
+rx_size_65_to_127_packets value:COUNTER:0:U
+rx_size_128_to_255_packets value:COUNTER:0:U
+rx_size_256_to_511_packets value:COUNTER:0:U
+rx_size_512_to_1023_packets value:COUNTER:0:U
+rx_size_1024_to_max_packets value:COUNTER:0:U
+rx_broadcast_packets value:COUNTER:0:U
+rx_multicast_packets value:COUNTER:0:U
+rx_fragment_errors value:COUNTER:0:U
+rx_undersize_errors value:COUNTER:0:U
+rx_oversize_errors value:COUNTER:0:U
+rx_jabber_errors value:COUNTER:0:U
+rx_management_packets value:COUNTER:0:U
+rx_management_dropped value:COUNTER:0:U
+tx_management_packets value:COUNTER:0:U
+rx_total_packets value:COUNTER:0:U
+rx_total_bytes value:COUNTER:0:U
+tx_total_packets value:COUNTER:0:U
+tx_size_64_packets value:COUNTER:0:U
+tx_size_65_to_127_packets value:COUNTER:0:U
+tx_size_128_to_255_packets value:COUNTER:0:U
+tx_size_256_to_511_packets value:COUNTER:0:U
+tx_size_512_to_1023_packets value:COUNTER:0:U
+tx_size_1024_to_max_packets value:COUNTER:0:U
+tx_multicast_packets value:COUNTER:0:U
+tx_broadcast_packets value:COUNTER:0:U
+rx_mac_short_packet_dropped value:COUNTER:0:U
+rx_l3_l4_xsum_error value:COUNTER:0:U
+flow_director_added_filters value:COUNTER:0:U
+flow_director_removed_filters value:COUNTER:0:U
+flow_director_filter_add_errors value:COUNTER:0:U
+flow_director_filter_remove_errors value:COUNTER:0:U
+flow_director_matched_filters value:COUNTER:0:U
+flow_director_missed_filters value:COUNTER:0:U
+rx_fcoe_crc_errors value:COUNTER:0:U
+rx_fcoe_dropped value:COUNTER:0:U
+rx_fcoe_mbuf_allocation_errors value:COUNTER:0:U
+rx_fcoe_packets value:COUNTER:0:U
+tx_fcoe_packets value:COUNTER:0:U
+rx_fcoe_bytes value:COUNTER:0:U
+tx_fcoe_bytes value:COUNTER:0:U
+rx_fcoe_no_direct_data_placement value:COUNTER:0:U
+rx_fcoe_no_direct_data_placement_ext_buff value:COUNTER:0:U
+tx_flow_control_xon_packets value:COUNTER:0:U
+rx_flow_control_xon_packets value:COUNTER:0:U
+tx_flow_control_xoff_packets value:COUNTER:0:U
+rx_flow_control_xoff_packets value:COUNTER:0:U
+rx_total_missed_packets value:COUNTER:0:U
+rx_priority0_mbuf_allocation_errors value:COUNTER:0:U
+rx_priority1_mbuf_allocation_errors value:COUNTER:0:U
+rx_priority2_mbuf_allocation_errors value:COUNTER:0:U
+rx_priority3_mbuf_allocation_errors value:COUNTER:0:U
+rx_priority4_mbuf_allocation_errors value:COUNTER:0:U
+rx_priority5_mbuf_allocation_errors value:COUNTER:0:U
+rx_priority6_mbuf_allocation_errors value:COUNTER:0:U
+rx_priority7_mbuf_allocation_errors value:COUNTER:0:U
+rx_priority0_dropped value:COUNTER:0:U
+rx_priority1_dropped value:COUNTER:0:U
+rx_priority2_dropped value:COUNTER:0:U
+rx_priority3_dropped value:COUNTER:0:U
+rx_priority4_dropped value:COUNTER:0:U
+rx_priority5_dropped value:COUNTER:0:U
+rx_priority6_dropped value:COUNTER:0:U
+rx_priority7_dropped value:COUNTER:0:U
+rx_priority0_xon_packets value:COUNTER:0:U
+rx_priority1_xon_packets value:COUNTER:0:U
+rx_priority2_xon_packets value:COUNTER:0:U
+rx_priority3_xon_packets value:COUNTER:0:U
+rx_priority4_xon_packets value:COUNTER:0:U
+rx_priority5_xon_packets value:COUNTER:0:U
+rx_priority6_xon_packets value:COUNTER:0:U
+rx_priority7_xon_packets value:COUNTER:0:U
+rx_priority0_xoff_packets value:COUNTER:0:U
+rx_priority1_xoff_packets value:COUNTER:0:U
+rx_priority2_xoff_packets value:COUNTER:0:U
+rx_priority3_xoff_packets value:COUNTER:0:U
+rx_priority4_xoff_packets value:COUNTER:0:U
+rx_priority5_xoff_packets value:COUNTER:0:U
+rx_priority6_xoff_packets value:COUNTER:0:U
+rx_priority7_xoff_packets value:COUNTER:0:U
+tx_priority0_xon_packets value:COUNTER:0:U
+tx_priority1_xon_packets value:COUNTER:0:U
+tx_priority2_xon_packets value:COUNTER:0:U
+tx_priority3_xon_packets value:COUNTER:0:U
+tx_priority4_xon_packets value:COUNTER:0:U
+tx_priority5_xon_packets value:COUNTER:0:U
+tx_priority6_xon_packets value:COUNTER:0:U
+tx_priority7_xon_packets value:COUNTER:0:U
+tx_priority0_xoff_packets value:COUNTER:0:U
+tx_priority1_xoff_packets value:COUNTER:0:U
+tx_priority2_xoff_packets value:COUNTER:0:U
+tx_priority3_xoff_packets value:COUNTER:0:U
+tx_priority4_xoff_packets value:COUNTER:0:U
+tx_priority5_xoff_packets value:COUNTER:0:U
+tx_priority6_xoff_packets value:COUNTER:0:U
+tx_priority7_xoff_packets value:COUNTER:0:U
+tx_priority0_xon_to_xoff_packets value:COUNTER:0:U
+tx_priority1_xon_to_xoff_packets value:COUNTER:0:U
+tx_priority2_xon_to_xoff_packets value:COUNTER:0:U
+tx_priority3_xon_to_xoff_packets value:COUNTER:0:U
+tx_priority4_xon_to_xoff_packets value:COUNTER:0:U
+tx_priority5_xon_to_xoff_packets value:COUNTER:0:U
+tx_priority6_xon_to_xoff_packets value:COUNTER:0:U
+tx_priority7_xon_to_xoff_packets value:COUNTER:0:U