CPViewer - visualize your cpview/cpinfo files in 5 minutes

[Petar_Markota]

What is CPViewer?

CPViewer is an open-source community tool which simplifies the way to get a very detailed visual insights into:

1) Exported cpview histories with charts related to CPU, memory, connections and packet rates, throughputs, RX&TX drops etc. using the visualization metrics tool called Grafana;

2) OS analysis (.info file) - healthcheck (sk121447) report for "known issues" and "things not to miss";

The tool comes within an .ova (exported VM) with everything already preconfigured.

The main benefits are:

• analyzing and identifying performance bottlenecks in minutes

• user-friendly: just upload it to web-site (no docker or other components needed) 

• Integrated OS analysis tool for richer results (healthcheck script)

• available to both customers and partners

CPViewer .ova file download - HERE.

CPViewer guide - HERE.

How to use it?

Video below explains all you need to do to get CPViewer up and running - 5 simple steps:

Written step by step guide:

1. Import the VM:

a. Download and import OVA image into your VMWare environment – DOWNLOAD LINK.

b. VM’s network adapter is set to NAT, it has IPv4 – 10.8.0.15, default gw – 10.8.0.2 and DNS – 8.8.8.8 predefined already, but you can adjust this by your needs;

c. Adjust your VMWare NAT adapter;

d. Credentials:
- OS: root/vpn123
- Grafana: admin/Vpn123!

*NOTE: Your VM must have internet access if you want to use GDrive download functionality.

2. Working with CPViewer portal:

a. CPViewer portal can process two types of data:

1) CPInfo files (contains cpinfo and cpview files) -> you will get 2 reports, grafana cpview insights and cpinfo OS analysis report (in separate tab);

2) CPView (.dat or .gz – with .dat in it) files only;

*NOTE: In case you are using type 1, please be aware that you need to either configure your browser to allow pop-ups for http://10.8.0.15:80 in order to get the CPInfo healthcheck report. Other option is just to go to http://10.8.0.15/healthcheck_reports manually and select a report you need.

b. After setting up the VM, open any browser and go to CPViewer portal -> http://10.8.0.15.

Select upload method:

1) Manual/attachment upload: you can submit .dat or .gz file (which contains .dat);

2) Google link (server will automatically download file from GDrive). In this case solid upload link is highly recommended;

*NOTE: You can pick one of the two methods, not both at the same time;

c. Enter customer`s name (this will be used for name db and datasource of cpview; d. Select version from which cpview was exported – R77.30 – R80.10 or R80.20+;

e. If you did all of the above, select the submit button and wait for your reports to get created;

*NOTE: Speed of the processing will depetend on the size of the file (upload time + querying/healthcheck.sh execution through the .dat/.info and taking all the relevant info).

3. After you get redirected to Grafana you will be able to see your cpview visualized through graphs focused on different parameters. In case you uploaded CPInfo file you will also get GW`s healthcheck report in a separate tab.

Few useful GrafanaUI details:

- Top left corner – selected datasource (datasources will automatically be deleted on weekly basis);

- On menu at the far left you will be able to see possible dashboards (do not need to be changed since everything related to your cpview is automatically provisioned);

- Top right corner – time span which we are looking into (this is also automatically set from the first to the last timestmmp from your cpview);

- When clicking on different views you will be able to adjust some parameters or queries according to your needs;

*NOTE: All datasources – their dbs and healthcheck reports are being automatically deleted every Monday at midnight. If you do not want this – enter crontab using command crontab – e from CLI, erase the camm of deleteALL.php and/or delete_hc.sh script/s and save it. 

*DISCLAIMER - This open source tool is provided “As Is”.  No representations or warranties are provided with the use of this tool.

[Zolo · Accepted solution]

It is because there are two "component_name" values (Host and LightSpeed) and the LightSpeed line may contains "0" (zero) data

For example:

#sqlite3 -noheader -csv cpview_services.dat 'SELECT * FROM main.fw_network_stats_traffic_rate limit 4'

1684095733,Host,87395,87593,318868012,329773658,762
1684095733,LightSpeed,0,0,0,0,0
1684095794,Host,86620,86796,306416968,317185350,758
1684095794,LightSpeed,0,0,0,0,0

Workaround (for example):

vi /var/www/html/bash/CPview_influx81.sh

Replace this line:

sqlite3 -noheader -csv $DB "SELECT inbound_throughput,outbound_throughput,inbound_packets_rate,outbound_packets_rate,conn_rate, Timestamp FROM main.fw_network_stats_traffic_rate" | gawk -F, '{printf "fw_counters inbound_throughput=%di,outbound_throughput=%di,inbound_packet_rate=%di,outbound_packet_rate=%di,conn_rate=%di %s\n",$1,$2,$3,$4,$5,$6;}' | curl -i -X POST "http://localhost:8086/write?db=$database_name&precision=s" --data-binary @-

To:

sqlite3 -noheader -csv $DB "SELECT inbound_throughput,outbound_throughput,inbound_packets_rate,outbound_packets_rate,conn_rate, Timestamp FROM main.fw_network_stats_traffic_rate where component_name='Host'" | gawk -F, '{printf "fw_counters inbound_throughput=%di,outbound_throughput=%di,inbound_packet_rate=%di,outbound_packet_rate=%di,conn_rate=%di %s\n",$1,$2,$3,$4,$5,$6;}' | curl -i -X POST "http://localhost:8086/write?db=$database_name&precision=s" --data-binary @-

Observe "where component_name='Host'" on the end of SELECT query...

[CheckPointerXL]

I got NO data for all the sections if i try to add the "WHERE component_name='Host'"

[Zolo]

Sorry I forget to say that you have to attach again (with another name) the cpview history database.

And you must select the R80.40+ option.

[CheckPointerXL]

I already tried, change db name, change customer name, reboot machine, but no luck. revert the original file fixed

my new line is:

sqlite3 -noheader -csv $DB "SELECT inbound_throughput,outbound_throughput,inbound_packets_rate,outbound_packets_rate,conn_rate, Timestamp FROM main.fw_network_stats_traffic_rate WHERE component_name='Host'" | gawk -F, '{printf "fw_counters inbound_throughput=%di,outbound_throughput=%di,inbound_packet_rate=%di,outbound_packet_rate=%di,conn_rate=%di %s\n",$1,$2,$3,$4,$5,$6;}' | curl -i -X POST "http://localhost:8086/write?db=$database_name&precision=s" --data-binary @-

[Zolo]

Try to copy cpview_services.dat to the server to be able to perform sql query on it.

And show me the output of this command:

sqlite3 -noheader -csv cpview_services.dat 'SELECT * FROM main.fw_network_stats_traffic_rate limit 4'

[CheckPointerXL]

i performed a query directly onto the fw:

sqlite3 -noheader -csv cpview_services.dat 'SELECT * FROM main.fw_network_stats_traffic_rate limit 4'
1684847752,Host,92464,99289,518523278,519025834,1002
1684847752,LightSpeed,0,0,0,0,0
1684847813,Host,75662,81760,402227954,417540566,938
1684847813,LightSpeed,0,0,0,0,0

could be the problem related to the fact that i upload the .gz file instead of .dat ?

[CheckPointerXL]

with .dat file it works !! thanks

 i guess that for the real throughput we should roughly take this data and multiple X2 , it is right?

[Zolo]

I don't think that it needs to be multiplied but I don't know for sure, sorry

[Wolfgang]

@CheckPointerXL why do you think you have to multiply the data, doesn't this reflect your real data ?

[CheckPointerXL]

Because throughput data it is divided in RX and TX. It seems that it doesnt reflet data obtained by throughput of single interfaces... i will check it better, thank you

[Pablo_Carchi]

Hello everybody.

I have uploaded a CPINFO file to CPViewer, from a Security Gateway R81.10 with VSX, installed on Open Server. The "Throughput" graph displayed a throughput very small (10 Mbps), but it is wrong, I believe that it draw data from VS-0. I have the following questions:

1. Is there any way to get throughput graph for each VS from CPViewer? Or the total throughput graph of the entire device?

2. In the "Interfaces throughput" graph, it displays and draw data for each interface, but it shows different measure unit (Mib), instead of Mbps (example 914 Mib). How should I interprete this information? Is there any way to translate from Mib to Mbps? And from Gib to Gbps?

	min	max	avg
RX_bit_"bond1"	14 Mib	308 Mib	72 Mib
RX_bit_"bond2"	54 Mib	435 Mib	160 Mib
RX_bit_"bond3"	353 Mib	2.847 Gib	1.110 Gib

Thanks for any suggest. Nice day to everyone.

[Wolfgang]

@Pablo_Carchi you have to switch to VSX context of the specific VS before running cpinfo. see The CPInfo utility 

Generating CPInfo on a VSX Gateway for a specific Virtual System

1. Connect to the command line on the VSX Gateway.
2. If your default shell is Gaia Clish, then go to the Expert mode:
   HostName> expert
3. Go to the context of the applicable Virtual System:
   [Expert@HostName:0]# vsenv <VS_ID>
4. Run the CPInfo tool (see "CLI Syntax").

for youre second question have alook at the older threat CPview Mbits vs Mbps columns 

[Pablo_Carchi]

Thank you so much @Wolfgang 

About the second answer, in the section "Interfaces Throughput", CPViewer displays the data with the measure unit "Mib" (Mebibit), not Mbits that will be Megabits. My question is how can I should interprete this measure unit (Mib) to obtain the data in Mbps.

I found this site (https://www.computerhope.com/jargon/m/mebibit.htm) that says that 1 Mib equals to 1'048.000 bit, so I believe that 1 Mib = 1,048 Mbits (or Mbps as CPview Mbits vs Mbps columns says). Do you think that it is a good conversion way?  

Thank you so much @Wolfgang 

[Wolfgang]

Yes, it is.