Diagnostics

Print version

8. Diagnostics

8.1. Overview

The Overview section serves to give general information about the M!DGE3.

8.1.1. Measurements

Section Overview – Measurements contains current data measurement (obtained from sensors).

  • Card Temperature – provides data about temperature (on CPU, modem).

  • Card Voltage – provides data about voltage measured on input connector.

Arrow-headed symbols (↑, ↓, →) have following meaning:

  • ↑ – Maximum-limit value. An alarm is triggered, when the value (displayed in brackets) is exceeded.

  • ↓ – Minimum-limit value. An alarm is triggered, when the value falls under the value, which is displayed in the brackets.

  • →- Value is supposed to head to another one.

[Note]Note

Value measurements are collected once per 10s.

8.1.2. Statistics

Section Overview – Statistics shows a short view of the statistics over last 15 minutes (from the time of opening the window or pressing Refresh button).

  • Cards Statistics are always displayed for all interfaces.

  • If the interface is off, its statistics (record) is displayed as “-“.

  • Statistics collection is updated every 1 s (each second is possible to see new values).

  • 15-min interval is collected by taking 14 mins from history + seconds passed from current minute.

8.2. Information

This section provides more detailed information (data extract) about settings of RipEX2 unit. It provides also a deeper explanation about some of set values and interfaces. Diagnostic data are provided as well.

8.2.1. Interfaces

Provides a complete information extract about all active interfaces (addresses, details and statistics included). All interfaces used by the linux router (including all internal interfaces like npi, loop, ag, ip6tnl, etc.) are displayed in this section.

Intefaces used in M!DGE3 units are in general either Bridged ports (BP-L2) or Routed interfaces (RI-L3).

All interfaces used by the linux router (internal interfaces excluded) are displayed in the following list.

if_<LanIface_Name>

LAN bridge interface RI-L3 type

(SETTINGS>Interfaces>Ethernet>Network interfaces)

if_<LanVlan_IfName>.<LanVlan_VlanId>

  • VLAN BP-L2 interface type (if used as a port in LAN bridge) (SETTINGS>Interfaces>Ethernet>Network interfaces>VLAN)

  • VLAN RI-L3 interface type (if not used as a port in LAN bridge) (SETTINGS>Interfaces>Ethernet>Network interfaces>IP/Subnet>VLAN)

eth1, eth2, eth3, eth4

Interface of physical Ethernet ports ETH1 – ETH4, BP-L2 interface type

eth0

Interface of physical port SFP (ETH5), BP-L2 interface type

wwan

Bridge interface of the Main cellular module, RI-L3 interface type (SETTINGS>Interface>Cellular>MAIN)

ext

Bridge interface of the EXT cellular module, , RI-L3 interface type (SETTINGS>Interface>Cellular>EXT)

gre_tap<INDEX>

GRE L2 tunnel interface, BP-L2 interface type (SETTINGS>VPN>GRE>L2)

gre_tun<INDEX >

GRE L3 tunnel interface, RI-L3 interface type (SETTINGS>VPN>GRE>L3)

lo

Loopback interface RI-L3 type of interface – The IP addresses of the loopback (ADVANCED>Interfaces>Loopback).

8.2.2. Routing

Provides information about data extract from section Routing

This section is divided into following parts:

  • System – complete data extract of system routing table. Displays data called by linux command “ip route show”.

  • Dynamic – complete data extract of internal routing table of dynamic routing service bird master4. Displays data called by linux command “birdcl show route all table master4”.

  • Babel – data extract of status of BABEL protocol. Displays data called by following linux commands: “birdcl show babel interfaces”, “birdcl show babel neighbors”, “birdcl show babel routes”, “birdcl show babel entries”, “birdcl show route all table babel_ipv4”.

  • OSPF- data extract of status of OSPF protocol. Displays data called by following linux commands: “birdcl show ospf neighbors”, “birdcl show ospf state”, “birdcl show ospf interface”, “birdcl show route all table ospf_ipv4”.

  • BGP – data extract of status of all BGP protocol instances. Displays data called by following linux commands: “birdcl show protocol ‘”bgp*”‘”, “birdcl show protocol all ‘”bgp*”‘”, “birdcl show route all table bgp_ipv4”.

8.2.3. Firewall

Provides general overview about data extract from sections L2, L3 and NAT.

8.2.3.1. Firewall L2

Displays data called by linux command “iptables -L”.

8.2.3.2. Firewall L3

Displays data called by following linux commands “iptables -nvL –line-numbers”.

8.2.3.3. NAT

Displays data called by following linux commands:

  • “iptables -t nat -nvL postrouting_user” – data about SNAT

  • “iptables -t nat -nvL prerouting_user” – data about DNAT

8.2.4. Quality of service

Creates a table about object and statistics extract for each given interface. This table contains:

  • Name of an interface

  • Status and statistics of front disciplines – displays data called by linux command “tc qdisc show”.

  • Status and statistics of classes – displays data called by linux command “tc class show”.

  • Status and statistics of filter – displays data called by linux command “tc filter show”.

8.2.5. Device

Provides general information about the unit (device).

8.2.5.1. System information

Basic unit information is provided.

  • Product code – Identifies the unit hardware.

  • Serial number – Unique unit identification number

  • FW version – Currently installed unit firmware

8.2.5.2. Advanced information

Additional unit information is provided which might be requested for advanced diagnostics. Partial description:

  • Modem board SN – Modem boards system number

  • CNF version – Version of the unit configuration.

  • Web client version – Version of the current web client

8.2.6. Diagnostic package

This menu serves for collecting data, either from local or remote station and storing them into a package (file). Diagnostic package serves primarily as a help tool, for RACOM’s technical support in case of any potential unit issues. Minimum size of a package is 5kB. Diagnostic package is downloaded already compressed, which saves approx. 1/3 of its original size.

Only one package collecting (applies for both local and remote) at a time is supported.

Package size

List box {Base; Verbose}, default = “Base”

Defines the size of the generated package.

Target

Defines the station, from which is the Diagnostic package being collected.

  • Diagnostic package from a local station – this parameter stays empty.

  • Diagnostic package from a remote station – destination IPv4 address of the requested station must be used.

Include
  • Configuration – configuration of the unit is added to the package (json format)

  • Event logs – adds a list of events exported to csv

    • Base: Last 50 events

    • Verbose: Last 500 events

  • Statistics – adds list of statistics exported to csv

    • Interval of frames statistics: 30 min

    • Base: 5 frames

    • Verbose: 10 frames

  • Status – lists detailed status of networks devices and services

  • System logs – adds last system logs

    • base: 100 of current lines from all logs

    • verbose: whole logs content

  • User credentials – adds a list of user accounts

After setting all parameters, click on “Generate” button. By clicking the “Refresh” button update the processing status of the package. Once the package is ready, it can be downloaded by clicking the “Download” button. After its download, the package is deleted from the unit. The package will be deleted even if its download is unsuccessful and if the download is not initiated, the package will be deleted automatically after 24h.

8.3. Events

This menu shows all events which occur within the unit history.

For filtering of events you can use the filtering tool. When no filter rules are used, the last 30 events will be displayed after Display button click.

Older events should be displayed using Load more button click, the events which occur during the viewing of this window can be loaded by using Load newer button.

Alarms are displayed in red color, warnings in orange, notices in black and debugs in gray.

It is possible to change severities of individual events in the menu SETTINGS/Device/Events.

Tab. 8.1: Default Events level description

Severity groupLevelSeverityColor codeDescriptionAction
ALARM0Emergency RedFaulty unit. HW repair is
probably needed.
Replace the unit. Contact Technical support.
1AlertRedUnit does not work. HW or SW problem.Check the unit. Consult Technical support.
2CriticalRedSerious error. Communication does not work.Check the unit immediately.
3ErrorRedError. Communication can work.Check the unit.
WARNING4WarningOrangeCommunication is OK. Self-healing action proceeded.When often, consult with Technical support.
5NoticeBlueSecurity important action
proceeded or I/O action.
Security check, the I/O status check.
INFO6InformationalInformational itemStandard behavior
7DebugDebug info, if set so.Debug

8.4. Statistics

M!DGE3 unit permanently monitors various system ‘channels’. There are several types of those channels: Physical interfaces (Ethernet ports, serial ports, MAIN, additional module interface (e.g. LTE module) when installed), virtual interfaces (e.g. VLAN interfaces) and HW sensors (CPU temperature, supply voltage, …). Monitored values are stored in the internal database.

Statistics page provides aggregated statistical data from this internal database. Data can be both displayed and downloaded in CSV format. This file format is suitable to be imported to any 3rd party spreadsheet program for further analysis.

There are two different options how to display statistics data:

Historical

Statistics counters are aggregated over the defined time interval. The interval is defined by two time stamps “From” and “To”.

Differential

Statistics counters are aggregated between the counter reset and the current time (the moment when the Display button was pressed). Reset is triggered by a unit reboot or by the Reset statistics button.

Reset statistics button – initiates the Differential statistic counters reset. Such a reset does not affect normal statistic counters – i.e. the Historical statistics are not affected by such a Reset at all.

Length of statistics data

Statistics data are stored in the internal database. There is a fixed memory size allocated for the statistics data – the database is limited by number of records. As a result of this, the length of statistics history – how old records are available – depends on the actual network configuration: The more monitored values, the higher the rate of new recorded values, the shorter the available history.

Some sets of monitored values are constant (Ethernet ports and their counters) or do not rise to a high values (COM ports, Terminal servers and their counters).

Statistics data in the context of unit interfaces

Fig. 8.1: Statistics data in the context of unit interfaces


8.4.1. Parameters

Statistics data are always retrieved as aggregated for a certain time Interval. This Interval can be set by putting specific date and time into “From” and “To” fields, or using buttons “Last day”, “Last hour” or “More options” fast presets (from several minutes to several days). Button “Set Current Time” sets current time to both From and To fields to ease current unit status diagnostics.

There are following sets of statistical data available in the unit:

  • Serial protocols statistics

  • Ethernet statistics

  • Cellular statistic

  • Measurements

“Display” button then shows chosen data below. “Download Selected Data” button generates CSV (UTF-8 encoded) file of all chosen systems’ data and downloads them as files without displaying them. Both “Display” and “Download …” buttons send a request for the required set of statistics data to the unit. Retrieving and transferring of the data takes some time. Downloading the data is practical when the user needs to process them in a spreadsheet and wants to save some bandwidth. It is also recommended to use spreadsheet editor like Microsoft Excel or Apple Numbers to process statistics on mobile devices due to better user experience provided by the specialized apps.

8.4.2. Serial protocol statistics

Serial protocols statistics provides set of data monitoring the COM port(s) and Terminal server (s). Only enabled interfaces are displayed. The statistics counters are based on packets entering or leaving the COM port or Terminal server module. As a result of this the ‘count’ values correspond to the Protocol messages (the “Protocol” selected on the specific COM port or Terminal server). If the packet is ‘glued’ from the several frames, it is evaluated as a single packet. In case of COM port statistics, the summary of ‘Correct’ and ‘Drop’ Bytes provides the total amount of Bytes on the physical interface.

Rx direction: from the connected (at the COM or ETH port) external device to the M!DGE3 unit (i.e. from the COM port module or Terminal server module to the Router module). Tx direction: from the M!DGE3 unit to the external device.

Interface – Interface name

Correct (Rx, Tx) – Correctly received / transmitted packets count and amount of data in Bytes. Accepted by the COM port or Terminal server module – based on the selected Protocol processing. Amount of data – for both Correct and Drop counters – is affected by COM port data only (i.e. IP headers of the UDP frames created in the COM port module are NOT counted).

Drop (Rx, Tx) – Dropped received / transmitted packets – reason: corrupted frame, CRC error, wrong protocol message, unsupported protocol message.

8.4.3. Ethernet statistics

Ethernet statistics provides set of data monitoring the physical Ethernet ports. Only enabled interfaces are displayed.

Only correctly received frames are handled. The counters correspond to the specific IP protocol types.

Rx direction: from the physical Ethernet port to the M!DGE3 unit (i.e. to the Router module). Tx direction: from the M!DGE3 unit to the physical Ethernet port.

Interface – Interface name.

UDP, TCP, ICMP, ARP, VLAN, Multicast – Packet count and amount of data in Bytes [B] for different protocol types – IPv4 traffic. Amount of data – for all counters – is summed over the whole Layer 2 Ethernet frame (i.e. all IP headers are counted).

IPv4 other – IPv4 traffic not handled by the previous counters

IPv6 – IPv6 traffic counter

Other – Counter summing up the frames which were not handled by the previous counters – for example MPLS and GOOSE protocols.

8.4.4. Cellular statistics

Cellular statistics are available for main cellular module and eventually for optional extension module if used.

8.4.4.1. Cellular interface statistics

Cellular interface statistics provides set of data collected from the interface between the Router module (IP routing engine in the unit) and the Cellular module. It corresponds to monitoring Cellular – Interface.

Tx direction: from the Router module to the Cellular module.
Rx direction: from the Cellular module to the Router module.

Interface

– “cell-main” interface is used for M!DGE3 MAIN cellular module.

– “cell-ext” interface is used for M!DGE3 optional extension cellular module.

UDP, TCP, ICMP, ARP

– Packet count and amount of data in Bytes [B] for different protocol types. Amount of data is summed over the whole Layer 2 Ethernet frame (i.e. all IP headers are counted).

IPv4 other

– Packets not handled by the previous counters (e.g. VLAN, services, GRE, IPsec (ESP), …).

IPv6

– IPv6 packets are handled separately.

Other

– other packets than IPv4 or IPv6

8.4.4.2. Cellular state statistics

Interface

– “cell-main” interface is used for M!DGE3 MAIN cellular module.

– “cell-ext” interface is used for M!DGE3 optional extension cellular module.

SIM [%]

– information about using the individual SIM cards during the time displayed in %.

Services [%]

– N/A (not available), 2G (e.g. GPRS, EDGE), 3G (e.g. UMTS), 4G (e.g. LTE) services usage displayed in % of time.

Networks [%]

– NC (not connected), Home (home network), Roaming (roaming network), Other (not matching previous type of networks) displayed in % of time.

[Note]Note

Values are rounded to an integer (in %).

8.4.4.3. Cellular signal statistics

Interface

– “cell-main” interface is used for M!DGE3 MAIN cellular module.

– “cell-ext” interface is used for M!DGE3 optional extension cellular module.

2G RSSI / 3G RSCP / 4G RSRP

– signal levels in dBm.

Samples

– number of samples used for the individual statistics.

avg / dev / min / max

– average / standard deviation / minimum / maximum value.

8.4.4.4. Measurements

Sensor

Measured values on M!DGE3.

count

Number of times that the sensor measured given value (counter).

avg / min / max

Average / minimum / maximum value.

8.5. Monitoring

Monitoring is an advanced on-line diagnostic tool, which enables a detailed analysis of communication over any of the M!DGE3 router interfaces. In addition to all the physical interfaces (MAIN, EXT, ETHs, COMs, TSs), some internal interfaces between software modules can be monitored when such advanced diagnostics is needed.

Interfaces

Fig. 8.2: Interfaces


Monitoring consists of two independent processes: settings of the monitored items and outputs. Please note that even if both of the outputs are switched off and some interfaces are set to On, the monitoring is still running in the background.

The monitoring screen has two main parts – Settings and Output

8.5.1. Settings

Save button – saves the new settings of the monitoring parameters.

Refresh monitoring data button – refreshes the settings menu according to the statistics status saved in the unit. The difference between the displayed and saved status can occur for example when the status is changed in different browser tab.

8.5.1.1. Overview

All status (On/Off) of individual interfaces are displayed on this place for quick overview on monitoring settings.

8.5.1.2. Interfaces

This section allows detailed settings of particular monitoring parameters for all interfaces.

Common parameters for several interfaces:

Rx enabled, Tx enabled

List box {On; Off}, default = “On”

A packet is considered a Tx one when it comes out from the respective software module (e.g. Terminal Server) and vice versa. When an external interface (e.g. Interface COM) is monitored, the Tx also means packets being transmitted from the M!DGE3 over the respective interface (Rx means “received”). Understanding the directions over the internal interfaces may not be that straightforward, please see Fig. 8.2, “Interfaces” above for clarification.

All

List box {On; Off}, default = “On”

Monitoring output can also be limited by IP protocol type. Select Off to be able to enable/disable specific protocol output individually – see next parameter(s).

UDP / TCP / ICMP / Other / ARP

List box {On; Off}, default = “Off”

Monitoring output of specific IP protocol limitation.

Offset [B]

Default = 0

Number of bytes from the beginning of packet/frame, which will not be displayed – the monitoring output is truncated by ‘Offset’ bytes at the beginning of the message.

Length [B]

Default = 32

Number of bytes to be displayed from each packet/frame.

Example: Offset=2, Length=4 means, that bytes from the 3rd byte to the 6th (inclusive) will be displayed:

Data (HEX):                 01AB 3798 A285 93CD 6B96

Monitoring output:                  3798 A285

Bandwidth

List box {LOW; NORMAL; HIGH; UNLIMITED}, default = “NORMAL”

Monitoring bandwidth limit to prevent overload of management link between client PC and the M!DGE3 unit. LOW (up to ~300 kb/s), NORMAL (up to ~800 kb/s), HIGH (up to ~2 Mb/s), UNLIMITED (up to ~8 Mb/s)

Source port (from) / Source port (to)

TCP/UDP source port to be enabled/disabled in the monitoring output. Use these parameters to specify the source range of ports <from – to>.

Destination port (from) / Destination port (to)

TCP/UDP destination port to be enabled/disabled in the monitoring output. Use these parameters to specify the destination range of ports <from – to>.

Dropped frames

List box {On; Off}, default = “Off”

When On, monitoring shows frames which are dropped (e.g. CRC is not valid, buffer overflow, …).

ETH interfaces

Include management traffic

List box {On; Off}, default = “Off”

Enable/disable management packets monitoring output.

Include ETH headers

List box {On; Off}, default = “Off”

Displays (enable) / omits (disable) L2 headers in the monitoring output.

Include reverse

List box {On; Off}, default = “Off”

Enable/disable reverse traffic (e.g. TCP reply to a request) monitoring.

Source IP / mask, Destination IP / mask

Monitoring output can also be limited to a specific address range – Source and Destination IP address and mask can be used to define the required range.

Cellular interfaces

Cinterion PLS83-W cellular module is connected to L3 layer, thus captured frames do not contain L2 header(s). For consistency reasons are following values added to the frame:

  • SRC mac: 0x0 0x0 0x0 0x0 0x0 0x0

  • Dst mac: 0x0 0x0 0x0 0x0 0x0 0x0

  • Ethertype: 0x0800

Cellular interface MAIN enabled (MAIN)

List box {On; Off}, default = “Off”

Cellular interface EXT enabled (EXT)

List box {On; Off}, default = “Off”

8.5.1.3. General

The settings of output parameters for file output – Max. file size and Time period, the first parameter matched closes the monitoring file. File is saved in compressed way, so the uncompressed and approximate compressed size is displayed in the list box.

Max. file size

List box {7 kB (~1 kB); 70 kB (~10 kB); 358 kB (~50 kB); 700 kB (~100 kB); 3 MB (~500 kB); 7 MB (~1 MB); max (~2 MB)}, default = “700 kB (~100 kB)”

Time period

List box {1 min; 2 min; 5 min; 10 min; 20 min; 30 min; 1 hour; 3 hours; 24 hours; Off}, default = “5 min”

Show time difference

List box {On; Off}, default = “Off”

When On, the time difference between subsequent packets is displayed in the monitoring output.

8.5.2. File output

Record / Stop recording button starts / stops recording to the file. The recording will be stopped immediately regardless of the size and time of recording. When the Record button is pressed for the second time the previously recorded data will be cleared.

Refresh button – refreshes the information about time remaining and size of the recorded data (in uncompressed way).

Download button – downloads file to a connected computer. The default name contains of the Unit name, date and time of the begin and day and time of the end of the monitoring. Before downloading you have to stop recording.

Clear button – allows to clear the monitoring data stored in the unit – both downloaded or not downloaded.

8.5.3. Console output

Monitor / Stop monitoring button

Download button – downloads the content of the console output as a file

Clear button – clears Console output screen

[Note]Note

If the amount of monitored data exceeds the limit (2.7 kB for remote monitoring and 32 kB for local monitoring) for one time period (approx. 1 s), some data will not be displayed in the console output. A note about the omited data will be inserted to the console output to the position of the non-displayed data.

8.6. Tools

Set of diagnostic tools

8.6.1. ICMP ping

All parameters used by standard ICMP ping are available. Start / Stop button starts / stops pinging.

8.6.2. RSS ping

RSS ping can be used for monitoring of the radio channel in case of hybrid networks (M!DGE3 / RipEX2 combination). In such networks, RSS ping comes through the whole network, but information about the RSS/MSE is evaluated only for radio hops.

RSS ping is a diagnostic tool for the radio performance measurement (Radio Signal Strength and modulation Mean Squared Error) of the individual radio hops within a M!DGE3 network. Hybrid networks are supported. Output format of different type (other than radio) of hops is similar to ICMP ping.

Destination IP

Destination IP address. This address must belong to a M!DGE3 unit as the RSS ping can be initiated only between two M!DGE3 units.

Length [B]

Number {8 – 1500}, default = 10

The length of data used by RSS ping. In case the length of RSS ping packet is longer than the length of Radio interface MTU, the first RSS ping packet will be lost and will cause decreasing of the packet length to the value matching to the current radio MTU. Random data are used as a payload.

Period [ms]

Number {100 – 3 600 000}, default = 1000

Period of sending RSS ping packets

When the period is set to a shorter number than the actual RTT, collisions might appear (depends on the selected Radio protocol). In order to reach the shortest possible period enable the Go on mode.

Timeout [ms]

Number {100 – 3 600 000}, default = 10000

Response timeout

Count

Number {1 – 10000}, default = 5

Number of RSS pings to be send

Source IP

The local IP address of M!DGE3 unit originating RSS ping. Blank field (equal to 0.0.0.0 address) is used to assign the source address automatically – address is assigned automatically according to the routing rules.

Go on

List box {On; Off}, default = “Off”

Go on mode. When Enabled, RSS pings are sent immediately after receiving the RSS ping reply (Period parameter is ignored).

Traces reserved

The RSS ping also contains data about the route (RSS, MSE), this parameter allows to set number of radio hops within the network to be measured. Radio hop is measured in both directions, so the number has to be higher than number of hops in route multiplied by 2 (for example: link consisting of 2 radio hops needs 5 traces to be reserved).

Output:

  • MC – Encodes Modulation and Coding – see transcription table:

    Tab. 8.2: Translation table for Modulation rates and FEC

     ModulationFEC
    002CPFSKFEC off
    01FEC 3/4
    104CPFSKFEC off
    11FEC 3/4
    80DPSKFEC off
    81FEC 3/4
    90pi/4 DQPSKFEC off
    91FEC 3/4
    A0D8PSKFEC off
    A1FEC 3/4
    B016DEQAMFEC off
    B1FEC 3/4
    C064QAMFEC off
    C1FEC 3/4
    D0FEC 5/6
    D1FEC 2/3
    E0256QAMFEC off
    E1FEC 3/4
    F0FEC 5/6
    F1FEC 2/3

  • RSS – Radio Signal Strength [dBm] – measured within the header reception

  • hMSE – Phy header modulation Mean Squared Error [dB] – measured within the header reception

  • dMSE – Data modulation Mean Squared Error [dB] – measured within the frame data part reception

8.6.3. Routing

Routing tool provides the next hop routing information of the given IP address.

Destination IP

The examined IP address.

Output

Output section provides the following details:

  • Examined address (example: 8.8.8.8)

  • Next hop (gateway) address (example: via 192.168.141.254)

  • Next hop interface (example: dev if_bridge)

  • Outgoing packet Source address (example: src 192.18.141.210)

8.6.4. System

Reboot button

Performs unit cold restart (power cycle equivalent).

8.7. Syslog

SYSLOG server IP

IP address of the remote Syslog server to which logs will be sent with severity higher than severity set in the Max. severity

SYSLOG server Port

Port used by the Syslog server

Max. severity

List box {Off; 0 Emergency; 1 Alert; 2 Critical; 3 Error}, default= ”Off”

Off – switches off the SYSLOG functionality

Only the events with set severity (and higher) will be sent to the Syslog server. Severities for individual Events can be set in Section 7.6.3, “Events”.

Login attempt

List box {Off; Web}, default = “Off”

Switches whether login attempts (both successful and unsuccessful) will be sent to the SYSLOG server.