Diagnostics

Print version

8. Diagnostics

8.1. Overview

The Overview section serves to give general information about the M!DGE3.

8.1.1. Measurements

Section Overview – Measurements contains current data measurement (obtained from sensors).

  • Card Temperature – provides data about temperature (on CPU, modem and radio).

    [Note]Note

    If the temperature exceeds its specified range (-45 – +95°C), the radio receiving/transmitting will be blocked. If the temperature will return to a specified range (-40 – +85°C), the radio starts receiving/transmitting again.

    This feature can be disabled in parameters Advanced > Interfaces > Radio > Radio parameters > Block radio in extreme temperatures.

  • Card Voltage – provides data about voltage measured on input connector.

Arrow-headed symbols (↑, ↓, →) have following meaning:

  • ↑ – Maximum-limit value. An alarm is triggered, when the value (displayed in brackets) is exceeded.

  • ↓ – Minimum-limit value. An alarm is triggered, when the value falls under the value, which is displayed in the brackets.

  • →- Value is supposed to head to another one.

[Note]Note

Value measurements are collected once per 10s (excluding radio interface, where values can be collected once per 10s or more often – depends on the radio communication character).

8.1.2. Statistics

Section Overview – Statistics shows a short view of the statistics over last 15 minutes (from the time of opening the window or pressing Refresh button).

  • Cards Statistics are always displayed for all interfaces.

  • If the interface is off, its statistics (record) is displayed as “-“.

  • Statistics collection is updated after 1s (each second is possible to see new values).

  • 15-min interval is collected by taking 14 mins from history + seconds passed from current minute.

8.2. Information

This section provides more detailed information (data extract) about settings of RipEX2 unit. It provides also a deeper explanation about some of set values and interfaces and. Diagnostic data are provided as well.

8.2.1. Interfaces

Provides a complete information extract about all active interfaces (addresses, details and statistics included). All interfaces used by the linux router (including all internal interfaces like npi, loop, ag, ip6tnl, etc.) are displayed in this section.

Intefaces used in M!DGE3 units are in general either Bridged ports (BP-L2) or Routed interfaces (RI-L3).

All interfaces used by the linux router (internal interfaces excluded) are displayed in the following list.

if_<LanIface_Name>

LAN bridge interface RI-L3 type

(SETTINGS>Interfaces>Ethernet>Network interfaces)

if_<LanVlan_IfName>.<LanVlan_VlanId>

  • VLAN BP-L2 interface type (if used as a port in LAN bridge) (SETTINGS>Interfaces>Ethernet>Network interfaces>VLAN)

  • VLAN RI-L3 interface type (if not used as a port in LAN bridge) (SETTINGS>Interfaces>Ethernet>Network interfaces>IP/Subnet>VLAN)

eth1, eth2, eth3, eth4

Interface of physical Ethernet ports ETH1 – ETH4, BP-L2 interface type

eth0

Interface of physical port SFP (ETH5), BP-L2 interface type

hstdby

Switched Hot Stand-by LAN interface – only in the active HS unit – used for traffic switched by HS unit, RI-L3 type of interface (SETTINGS

radio

RF radio interface

  • BP-L2 interface type when Bridge mode of Radio interface is used (SETTINGS>Interface>Radio)

  • BP-L2 interface type when Bridge mode of Radio interface is used (SETTINGS>Interface>Radio)

Wwan

Bridge interface of the Main cellular module, RI-L3 interface type (SETTINGS>Interface>Cellular>MAIN)

Aux

Bridge interface of the EXT cellular module, , RI-L3 interface type (SETTINGS>Interface>Cellular>/EXT)

gre_tap<INDEX>

GRE L2 tunnel interface, BP-L2 interface type (SETTINGS>VPN>GRE>L2)

gre_tun<INDEX >

GRE L3 tunnel interface, RI-L3 interface type (SETTINGS>VPN>GRE>L3)

lo

Loopback interface RI-L3 type of interface – The IP addresses of the loopback (ADVANCED>Interfaces>Loopback).

8.2.2. Routing

Provides information about data extract from section Routing

This section is divided into following parts:

  • Static – complete data extract of system routing table. Displays data called by linux command “ip route show”.

  • Dynamic – complete data extract of internal routing table of dynamic routing service bird master4. Displays data called by linux command “birdcl show route all table master4”.

  • Babel – data extract of status of BABEL protocol. Displays data called by following linux commands: “birdcl show babel interfaces”, “birdcl show babel neighbors”, “birdcl show babel routes”, “birdcl show babel entries”, “birdcl show route all table babel_ipv4”.

  • OSPF- data extract of status of OSPF protocol. Displays data called by following linux commands: “birdcl show ospf neighbors”, “birdcl show ospf state”, “birdcl show ospf interface”, “birdcl show route all table ospf_ipv4”.

  • BGP – data extract of status of all BGP protocol instances. Displays data called by following linux commands: “birdcl show protocol ‘”bgp*”‘”, “birdcl show protocol all ‘”bgp*”‘”, “birdcl show route all table bgp_ipv4”.

8.2.3. Firewall

Provides general overview about data extract from sections L2, L3 and NAT.

8.2.3.1. Firewall L2

Displays data called by linux command “iptables -L”.

8.2.3.2. Firewall L3

Displays data called by following linux commands “iptables -nvL –line-numbers”.

8.2.3.3. NAT

Displays data called by following linux commands:

“iptables” – displays user routing rules

“iptables -t nat -nvL postrouting_user” – data about SNAT

“iptables -t nat -nvL prerouting_user” – data about DNAT

“conntrack -L” – connections and its status

8.2.4. Quality of service

Creates a table about object and statistics extract for each given interface. This table contains:

  • Name of an interface

  • Status and statistics of front disciplines – displays data called by linux command “tc qdisc show”.

  • Status and statistics of classes – displays data called by linux command “tc class show”.

  • Status and statistics of filter – displays data called by linux command “tc filter show”.

8.2.5. Device

Provides general information about the unit (device).

8.2.5.1. System information

Basic unit information is provided.

  • Product code – Identifies the unit hardware.

  • Serial number – Unique unit identification number

  • FW version – Currently installed unit firmware

8.2.5.2. Advanced information

Additional unit information is provided which might be requested for advanced diagnostics. Partial description:

  • Modern board SN – Modern boards system number

  • CNF version – Version of the unit configuration. Configuration version is updated if the new firmware version brings major configuration changes leading to incompatibility with the previous configuration. See Section 7.6.2, “Configuration” for more details.

  • Web client version – Version of the current web client

  • Region ID – If the unit is delivered with certain limits applied according to a specific Region (specified when ordering a new unit), the specific Region identification is visible here. No limits are applied if the Region ID is empty (“—”).

  • Region description – Detailed description of the Region (if active).

8.2.6. Diagnostic package

This menu serves for collecting data, either from local or remote station and storing them into a package (file). Diagnostic package serves primarily as a help tool, for RACOM’s technical support in case of any potential unit issues. Minimum size of a package is 5kB. Maximum size depends on the amount of radio links contained in the statistics.

Only one package collecting (applies for both local and remote) at a time is supported.

Package size

List box {Base; Verbose}, default = “Base”

Defines the size of the generated package.

Target

Defines the station, from which is the Diagnostic package being collected.

  • Diagnostic package from a local station – this parameter stays empty.

  • Diagnostic package from a remote station – destination IPv4 address of the requested station must be used.

Include
  • Configuration – configuration of the unit is added to the package (json format)

  • Event logs – adds a list of events exported to csv

    • Base: Last 50 events

    • Verbose: Last 500 events

  • Statistics – adds list of statistics exported to csv

    • Interval of frames statistics: 30 min

    • Base: 5 frames

    • Verbose: 10 frames

  • Status – lists detailed status of networks devices and services

  • System logs – adds last system logs

    • base: 100 of current lines from all logs

    • verbose: whole logs content

  • User credentials – adds a list of user accounts

After setting all parameters, click on “Generate” button. By occasionally clicking the “Refresh” button it is possible to view the status of the package. Once the package is ready, it can be downloaded by clicking the “Download” button. After its download, the package is deleted. The package will be deleted even if its download is unsuccessful. If the download is not initiated, the package will be automatically deleted after 24h.

8.3. Events

This menu shows all events which occur within the unit history.

For filtering of events you can use the filtering tool. When no filter rules are used, the last 30 events will be displayed after Display button click.

Older events should be displayed using Load more button click, the events which occur during the viewing of this window can be loaded by using Load newer button.

Alarms are displayed in red color, warnings in orange, notices in black and debugs in gray.

It is possible to change severities of individual events in the menu SETTINGS/Device/Events.

Tab. 8.1: Default Events level description

Severity groupLevelNameColor codeDescriptionAction
ALARM0Emergency RedFaulty unit. HW repair is
probably needed.
Replace the unit. Contact Technical support.
1AlertRedUnit does not work. HW or SW problem.Check the unit. Consult Technical support.
2CriticalRedSerious error. Communication does not work.Check the unit immediately.
3ErrorRedError. Communication can work.Check the unit.
WARNING4WarningOrangeCommunication is OK. Self-healing action proceeded.When often consult with Technical support.
5NoticeOrangeSecurity important action
proceeded or I/O action.
Security check, the I/O status check.
INFO6InformationalBlueInformational itemStandard behavior
7DebugGreyDebug info, if set so.Debug

8.4. Statistics

M!DGE3 unit permanently monitors various system ‘channels’. There are several types of those channels: Physical interfaces (Ethernet ports, serial ports, radio interface, additional module interface (e.g. LTE module) when installed), virtual interfaces (e.g. VLAN interfaces) and HW sensors (CPU temperature, supply voltage, …). Monitored values are stored in the internal database.

Statistics page provides aggregated statistical data from this internal database. Data can be both displayed and downloaded in CSV format. This file format is suitable to be imported to any 3rd party spreadsheet program for further analysis.

There are two different options how to display statistics data:

Historical

Statistics counters are aggregated over the defined time interval. The interval is defined by two time stamps “From” and “To”.

Differential

Statistics counters are aggregated between the counter reset and the current time (the moment when the Display button was pressed). Reset is triggered by a unit reboot or by the Reset statistics button.

Reset statistics button – initiates the Differential statistic counters reset. Such a reset does not affect normal statistic counters – i.e. the Historical statistics is not affected by such a Reset at all.

Length of statistics data

Statistics data are stored in the internal database. There is a fixed memory size allocated for the statistics data – the database is limited by number of records. As a result of this, the length of statistics history – how old records are available – depends on the actual network configuration: The more monitored values, the higher the rate of new recorded values, the shorter the available history.

Some sets of monitored values are constant (Ethernet ports and their counters) or do not rise to a high values (COM ports, Terminal servers and their counters). What affects the length of available history the most is the number of radio links – how many radio “neighbours” the unit communicates with or how many of the neighbouring signals are received. This is different for each unit in the network. For example Base station in a network operating Base driven protocol is the most active unit thus it has the shortest history available.

Statistics data in the context of unit interfaces

Fig. 8.1: Statistics data in the context of unit interfaces


8.4.1. Parameters

Statistics data are always retrieved as aggregated for a certain time Interval. This Interval can be set by putting specific date and time into “From” and “To” fields, or using buttons “Last day”, “Last hour” or “More options” fast presets (from several minutes to several days). Button “Set Current Time” sets current time to both From and To fields to ease current unit status diagnostics.

There are following sets of statistical data available in the unit:

  • Serial protocols statistics

  • Ethernet statistics

  • Cellular statistic (if cellular interface is available)

“Display” button then shows chosen data below. “Download Selected Data” button generates CSV (UTF-8 encoded) file of all chosen systems’ data and downloads them as files without displaying them. Both “Display” and “Download …” buttons send a request for the required set of statistics data to the unit. Retrieving and transferring of the data (over the radio channel) takes some time. Downloading the data is practical when the user needs to process them in a spreadsheet and wants to save some bandwidth. It is also recommended to use spreadsheet editor like Microsoft Excel or Apple Numbers to process statistics on mobile devices due to better user experience provided by the specialized apps.

8.4.2. Serial protocol statistics

Serial protocols statistics provides set of data monitoring the COM port(s) and Terminal server (s). Only enabled interfaces are displayed. The statistics counters are based on packets entering or leaving the COM port or Terminal server module. As a result of this the ‘count’ values correspond to the Protocol messages (the “Protocol” selected on the specific COM port or Terminal server). If the packet is ‘glued’ from the several frames, it is evaluated as a single packet. In case of COM port statistic, the summary of ‘Correct’ and ‘Drop’ Bytes provides the total amount of Bytes on the physical interface.

Rx direction: from the connected (at the COM or ETH port) external device to the M!DGE3 unit (i.e. from the COM port module or Terminal server module to the Router module). Tx direction: from the M!DGE3 unit to the external device.

Interface – Interface name

Correct (Rx, Tx) – Correctly received / transmitted packets count and amount of data in Bytes. Accepted by the COM port or Terminal server module – based on the selected Protocol processing. Amount of data – for both Correct and Drop counters – is affected by COM port data only (i.e. IP headers of the UDP frames created in the COM port module are NOT counted).

Drop (Rx, Tx) – Dropped received / transmitted packets – reason: corrupted frame, CRC error, wrong protocol message, unsupported protocol message.

8.4.3. Ethernet statistics

Ethernet statistics provides set of data monitoring the physical Ethernet ports. Only enabled interfaces are displayed.

Only correctly received frames are handled. The counters correspond to the specific IP protocol types.

Rx direction: from the physical Ethernet port to the M!DGE3 unit (i.e. to the Router module). Tx direction: from the M!DGE3 unit to the physical Ethernet port.

Interface – Interface name.

UDP, TCP, ICMP, ARP, VLAN, Multicast – Packet count and amount of data in Bytes [B] for different protocol types – IPv4 traffic. Amount of data – for all counters – is summed over the whole Layer 2 Ethernet frame (i.e. all IP headers are counted).

IPv4 other – IPv4 traffic not handled by the previous counters

IPv6 – IPv6 traffic counter

Other – Counter summing up the frames which were not handled by the previous counters – for example MPLS and GOOSE protocols.

8.4.4. Cellular statistics

8.4.4.1. Cellular interface statistics

Cellular interface statistics provides set of data collected from the interface between the Router module (IP routing engine in the unit) and the Cellular module. It corresponds to monitoring Cellular – Interface.

Tx direction: from the Router module to the Cellular module.
Rx direction: from the Cellular module to the Router module.

Cell-aux

– internal interface name. All information is displayed for this interface together.

UDP, TCP, ICMP, ARP

– Packet count and amount of data in Bytes [B] for different protocol types. Amount of data is summed over the whole Layer 2 Ethernet frame (i.e. all IP headers are counted).

IPv4 other

– Packets not handled by the previous counters (e.g. VLAN, services, GRE, IPsec (ESP), …).

IPv6

– IPv6 packets are handled separately.

Other

– other packets than IPv4 or IPv6

8.4.4.2. Cellular state statistics

Interface

– “Cell-aux” interface is used for M!DGE3 cellular module.

SIM [%]

– information about using the individual SIM cards during the time displayed in %.

Services [%]

– N/A (not available), 2G (GPRS, EDGE), 3G (UMTS), 4G (LTE) services usage displayed in % of time.

Networks [%]

– NC (not connected), Home (home network), Roaming (roaming network), Other (not matching previous type of networks) displayed in % of time.

[Note]Note

Values are rounded to an integer (in %).

8.4.4.3. Cellular signal statistics

Interface

– “Cell-aux” interface is used for M!DGE3 cellular module.

2G RSSI / 3G RSCP / 4G RSRP

– signal levels in dBm.

Samples

– number of samples used for the individual statistics.

avg / dev / min / max

– average / standard deviation / minimum / maximum value.

8.5. Monitoring

Monitoring is an advanced on-line diagnostic tool, which enables a detailed analysis of communication over any of the M!DGE3 router interfaces. In addition to all the physical interfaces (RADIO, ETHs, COMs, TSs), some internal interfaces between software modules can be monitored when such advanced diagnostics is needed.

Interfaces

Fig. 8.2: Interfaces


Monitoring consists of two independent processes: settings of the monitored items and outputs. Please note that even if both of the outputs are switched off and some interfaces are set to On the monitoring is still running in the background.

The monitoring screen accordingly has two main parts – Setting and Output

8.5.1. Settings

Save button – saves the new settings of the monitoring parameters.

Refresh monitoring data button – refreshes the settings menu according to the statistics status saved in the unit. The difference between the displayed and saved status can occur for example when the status is changed in different browser tab.

8.5.1.1. Overview

All status (On/Off) of individual interfaces are displayed on this place for quick overview on monitoring settings.

8.5.1.2. Interfaces

This section allows detailed settings of very particular monitoring parameters for all individual interfaces.

Common parameters for several interfaces:

Rx enabled, Tx enabled

List box {On; Off}, default = “On”

A packet is considered a Tx one when it comes out from the respective software module (e.g. RADIO or Terminal Server) and vice versa. When an external interface (e.g. Interface COM) is monitored, the Tx also means packets being transmitted from the M!DGE3 over the respective interface (Rx means “received”). Understanding the directions over the internal interfaces may not be that straightforward, please consult ??? above for clarification.

All

List box {On; Off}, default = “On”

Monitoring output can also be limited by IP protocol type. Select Off to be able to enable/disable specific protocol output individually – see next parameter(s).

UDP / TCP / ICMP / Other / ARP

List box {On; Off}, default = “Off”

Monitoring output of specific IP protocol limitation.

Offset [B]

Default = 0

Number of bytes from the beginning of packet/frame, which will not be displayed – the monitoring output is truncated by ‘Offset’ bytes at the beginning of the message.

Length [B]

Default = 32

Number of bytes to be displayed from each packet/frame.

Example: Offset=2, Length=4 means, that bytes from the 3rd byte to the 6th (inclusive) will be displayed:

Data (HEX):                 01AB 3798 A285 93CD 6B96

Monitoring output:                  3798 A285

Bandwidth

List box {LOW; NORMAL; HIGH; UNLIMITED}, default = “NORMAL”

Monitoring bandwidth limit to prevent overload of management link between client PC and the M!DGE3 unit. LOW (up to ~300 kb/s), NORMAL (up to ~800 kb/s), HIGH (up to ~2 Mb/s), UNLIMITED (up to ~8 Mb/s)

Source port (from) (to)

TCP/UDP source port to be enabled/disabled in the monitoring output. Use “… (to)” parameter to specify range of ports <from – to>.

Destination port (from) (to)

TCP/UDP destination port to be enabled/disabled in the monitoring output. Use “… (to)” parameter to specify range of ports <from – to>.

Dropped frames

List box {On; Off}, default = “Off”

When On, monitoring shows packets which are dropped (e.g. CRC is not valid, buffer overflow, …).

ETH interfaces

Include management traffic

List box {On; Off}, default = “Off”

Enable/disable management packets monitoring output.

Include ETH headers

List box {On; Off}, default = “Off”

Displays (enable) / omits (disable) L2 headers in the monitoring output

Include reverse

List box {On; Off}, default = “Off”

Enable/disable reverse traffic (e.g. TCP reply to a request) monitoring.

Source IP / mask, Destination IP / mask

Monitoring output can also be limited to a specific address range – Source and Destination IP address and mask can be used to define the required range.

8.5.1.3. General

The settings of output parameters for file output – Max. file size and Time period, the first parameter matched closes the monitoring file. File is saved in compressed way, so the uncompressed and approximate compressed size is displayed in the list box.

Max. file size

List box {7 kB (~1 kB); 70 kB (~10 kB); 358 kB (~50 kB); 700 kB (~100 kB); 3 MB (~500 kB); 7 MB (~1 MB); max (~2 MB)}, default = “700 kB (~100 kB)”

Time period

List box {1 min; 2 min; 5 min; 10 min; 20 min; 30 min; 1 hour; 3 hours; 24 hours; Off}, default = “5 min”

Show time difference

List box {On; Off}, default = “Off”

When On, the time difference between subsequent packets is displayed in the monitoring output.

8.5.2. File output

Record / Stop recording button starts / stops recording to the file. The recording will be stopped immediately regardless of the size and time of recording. When the Record button is pressed for the second time the previously recorded data will be cleared.

Refresh button – refreshes the information about time remaining and size of the recorded data (in uncompressed way).

Download button – downloads file to a connected computer. The default name contains of the Unit name, date and time of the begin and day and time of the end of the monitoring. Before downloading you have to stop recording.

Clear button – allows to clear the monitoring data stored in the unit – both downloaded or un-downloaded.

8.5.3. Console output

Monitor / Stop monitoring button

Download button – downloads the content of the console output as a file

Clear button – clears Console output screen

[Note]Note

If the amount of monitored data exceeds the limit (2.7 kB for remote monitoring and 32 kB for local monitoring) for one time period (approx. 1 s), some data will not be displayed in the console output. A note about the omited data will be inserted to the console output to the position of the non-displayed data.

8.6. Routing

Displays all routing rules used in unit. Section Statics contains all static rules, Dynamic all currently used dynamic rules, BABEL, OSPF and BGP shows all rules set for individual dynamic protocols.

8.7. Tools

Set of diagnostic tools

8.7.1. ICMP ping

All parameters used by standard ICMp ping are available. Start / Stop button starts / stops pinging.

8.7.2. Routing

Routing tool provides the next hop routing information of the given IP address.

Destination IP

The examined IP address.

Output

Output section provides the following details:

  • Examined address (example: 8.8.8.8)

  • Next hop (gateway) address (example: via 192.168.141.254)

  • Next hop interface (example: dev if_bridge)

  • Outgoing packet Source address (example: src 192.18.141.210)

8.7.3. System

Reboot button

Performs unit cold restart (power cycle equivalent).

8.8. Syslog

SYSLOG server IP

IP address of the remote Syslog server to which will be send logs with severity higher than severity set in the Max. severity

SYSTOG server Port

Port used by the Syslog server

Max. severity

List box {Off; 0 Emergency; 1 Alert; 2 Critical; 3 Error}, default= ”Off”

Off – switched off the SYSLOG functionality

Only the events with set severity (and higher) will be send to the Syslog server. Severities for individual Events can be set in menu ADVANCED/Device/Events/Events

Login attempt

List box {On; Off}, default = “Off”

Switches whether login attempts (both successful and unsuccessful) will be sent to the SYSLOG server.