Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Petar_Markota
Employee Employee
Employee
Jump to solution

CPViewer - visualize your cpview/cpinfo files in 5 minutes

 

What is CPViewer?

CPViewer is an open-source community tool which simplifies the way to get a very detailed visual insights into:

1) Exported cpview histories with charts related to CPU, memory, connections and packet rates, throughputs, RX&TX drops etc. using the visualization metrics tool called Grafana;

2) OS analysis (.info file) - healthcheck (sk121447) report for "known issues" and "things not to miss";

The tool comes within an .ova (exported VM) with everything already preconfigured.

 

The main benefits are:

  • analyzing and identifying performance bottlenecks in minutes

  • user-friendly: just upload it to web-site (no docker or other components needed) 

  • Integrated OS analysis tool for richer results (healthcheck script)

  • available to both customers and partners

 

CPViewer .ova file download - HERE.

 

CPViewer guide - HERE.

 

How to use it?

 

Video below explains all you need to do to get CPViewer up and running - 5 simple steps:

 

 

Written step by step guide:

 

1. Import the VM:

a. Download and import OVA image into your VMWare environment – DOWNLOAD LINK.

b. VM’s network adapter is set to NAT, it has IPv4 – 10.8.0.15, default gw – 10.8.0.2 and DNS – 8.8.8.8 predefined already, but you can adjust this by your needs;

c. Adjust your VMWare NAT adapter;

d. Credentials:
- OS: root/vpn123
- Grafana: admin/Vpn123!

*NOTE: Your VM must have internet access if you want to use GDrive download functionality.

 

2. Working with CPViewer portal:

a. CPViewer portal can process two types of data:

1) CPInfo files (contains cpinfo and cpview files) -> you will get 2 reports, grafana cpview insights and cpinfo OS analysis report (in separate tab);

2) CPView (.dat or .gz – with .dat in it) files only;

*NOTE: In case you are using type 1, please be aware that you need to either configure your browser to allow pop-ups for http://10.8.0.15:80 in order to get the CPInfo healthcheck report. Other option is just to go to http://10.8.0.15/healthcheck_reports manually and select a report you need.

b. After setting up the VM, open any browser and go to CPViewer portal -> http://10.8.0.15.

Select upload method:

1) Manual/attachment upload: you can submit .dat or .gz file (which contains .dat);

2) Google link (server will automatically download file from GDrive). In this case solid upload link is highly recommended;

*NOTE: You can pick one of the two methods, not both at the same time;

c. Enter customer`s name (this will be used for name db and datasource of cpview; d. Select version from which cpview was exported – R77.30 – R80.10 or R80.20+;

e. If you did all of the above, select the submit button and wait for your reports to get created;

*NOTE: Speed of the processing will depetend on the size of the file (upload time + querying/healthcheck.sh execution through the .dat/.info and taking all the relevant info).



3. After you get redirected to Grafana you will be able to see your cpview visualized through graphs focused on different parameters. In case you uploaded CPInfo file you will also get GW`s healthcheck report in a separate tab.

Few useful GrafanaUI details:

- Top left corner – selected datasource (datasources will automatically be deleted on weekly basis);

- On menu at the far left you will be able to see possible dashboards (do not need to be changed since everything related to your cpview is automatically provisioned);

- Top right corner – time span which we are looking into (this is also automatically set from the first to the last timestmmp from your cpview);

- When clicking on different views you will be able to adjust some parameters or queries according to your needs;

*NOTE: All datasources – their dbs and healthcheck reports are being automatically deleted every Monday at midnight. If you do not want this – enter crontab using command crontab – e from CLI, erase the camm of deleteALL.php and/or delete_hc.sh script/s and save it. 

 

*DISCLAIMER - This open source tool is provided “As Is”.  No representations or warranties are provided with the use of this tool.

(1)
3 Solutions

Accepted Solutions
_Val_
Admin
Admin

Will that help?

https://vitux.com/ubuntu-network-configuration/

Web service most probably should also be restarted though.

View solution in original post

Danny
Champion Champion
Champion

CPViewer has been advanced into Skyline (TechTalk, Toolbox).

View solution in original post

Zolo
Contributor
Contributor

It is because there are two "component_name" values (Host and LightSpeed) and the LightSpeed line may contains "0" (zero) data

For example:

#sqlite3 -noheader -csv cpview_services.dat 'SELECT * FROM main.fw_network_stats_traffic_rate limit 4'

1684095733,Host,87395,87593,318868012,329773658,762
1684095733,LightSpeed,0,0,0,0,0
1684095794,Host,86620,86796,306416968,317185350,758
1684095794,LightSpeed,0,0,0,0,0

Workaround (for example):

vi /var/www/html/bash/CPview_influx81.sh

Replace this line:

sqlite3 -noheader -csv $DB "SELECT inbound_throughput,outbound_throughput,inbound_packets_rate,outbound_packets_rate,conn_rate, Timestamp FROM main.fw_network_stats_traffic_rate" | gawk -F, '{printf "fw_counters inbound_throughput=%di,outbound_throughput=%di,inbound_packet_rate=%di,outbound_packet_rate=%di,conn_rate=%di %s\n",$1,$2,$3,$4,$5,$6;}' | curl -i -X POST "http://localhost:8086/write?db=$database_name&precision=s" --data-binary @-

To:

sqlite3 -noheader -csv $DB "SELECT inbound_throughput,outbound_throughput,inbound_packets_rate,outbound_packets_rate,conn_rate, Timestamp FROM main.fw_network_stats_traffic_rate where component_name='Host'" | gawk -F, '{printf "fw_counters inbound_throughput=%di,outbound_throughput=%di,inbound_packet_rate=%di,outbound_packet_rate=%di,conn_rate=%di %s\n",$1,$2,$3,$4,$5,$6;}' | curl -i -X POST "http://localhost:8086/write?db=$database_name&precision=s" --data-binary @-

Observe "where component_name='Host'" on the end of SELECT query...

View solution in original post

103 Replies
alexsp
Employee Alumnus
Employee Alumnus
Amazing tool and very helpful!
PhoneBoy
Admin
Admin
And clearly to customers if you're posting on CheckMates.
Nicely done!
JesusJ_Sanz
Explorer

Thank you very much for sharing!

Very intesresting tool for support purposes. Nicely done! 🙂

CCSA CCSE
_Val_
Admin
Admin

Great job, @Petar_Markota!

Milos_Jovovic
Contributor

Well done Petar! Svaka cast. 🙂

Danny
Champion Champion
Champion

Thanks for reminding us that the CPview history database can be visualized with tools like Grafana, DBPlot and others.

The web visualization output of Check Points Gaia Healthcheck script is a default feature of the tool with full vendor support. Within your project one would lose the vendor support, right?

A few other questions pop up in my head:

  • You publish Check Points copyrighted health check script publicly on a Google drive (within an ova file) making it downloadable for everyone thus circumventing Check Points Support Center Access policy -> Where is Check Points statement that they granted permission for this?
  • You use Check Points official document template including copyright notes while you distribute Open Source software -> What about copyrights?
  • Your CPViewer guide looks like an official Check Point document but Check Point doesn't know anything about it?
  • Your CPViewer guide is entitled for SEs/partners, however this thread says it's for customers as well. What is correct?
  • Your CPViewer guide says: *NOTE: Your VM must have internet access.. Why?
  • Your CPViewer accepts downloads of Check Point CPinfo files from Google Drives. Why would anyone ever upload his confidential Check Point data to a Google drive?
  • You are a Check Point employee. Why aren't you hosting this on Check Points GitHub account? Check Point has other open source projects that are hosted there (example: sk120342). This would look way more official than the Google drive solution.
Petar_Markota
Employee Employee
Employee

Hi @Danny ,

Thanks for feedback and great observations.

Prior to publishing CPViewer on Check Mates (for everyone) we checked with product management and legal department after which we got an approval for the public post on here. Months before, this tool was available internally/was shared with partners and after a great feedback we decided to publish it for all since it obviously can be helpful to anyone with CP gateways and it does not contain anything that is confidential.

HC script will be updated in case of a new versions.


Answers to your questions:

1) The main idea of the tool is to have it all centralized in one place, automatically provisioned, where without much hustle people can use its capabilities and get the reports ASAP when needed. Healthcheck script is a great feature which acts as an addition to cpview chart and enriches the info we get, so it was a nice idea to add it up to the tool. Again, the purpose of the tool is only to help people solve potential problems faster and easier, so I don`t think we need to be so strict on this.
2,3) I was not notified about the word template issue which you mentioned. Once again, thanks for great observation. It is removed now.
4) I renamed it.
5) The tool has a feature where you can add a GDrive link to your cpview/cpinfo files from where they would be downloaded. This is the only reason why it needs to have internet access. If you will be attaching files only from your local machine you don`t need Internet access. I added a clarification.
6) Here, you answered your question from above. Some people do, it`s just another feature which can be used if you want.
7) We went through the process of approval for the tool`s posting on CheckMates and I was not pointed to use any of those. Thank you for mentioning it though, we might transfer it to one of them in the future.

AkosBakos
Collaborator

Hi Petar,

I used a lof of times this tool, it is amazing to visualize the heavily loaded interfaces.

It helped me a lot of times.

Now I have a question. Does the tool support R81.10 JFH66?

My experience is that, the overal throughput was not shown correctly, I see only a constant line.

Looking forward to your answer,

Akos

HenriqueFM96
Employee Alumnus
Employee Alumnus

Hey @Danny ,

Glad to hear from you!

I really think this tool is helping a lot of people so far, both employees and not.
@Petar_Markota  might be able to tell us how many people already thanked him so far...
Apart from that, just have a look at the previous comments on this post, many Check Point employees are really happy with the tool.

Just trying to answer some of your questions...
- By using this tool, you don't lose your product support. As the VM doesn't need to be in your network, it can be in a lab/monitoring environment. So you just need to import the files to populate the Grafana with the data from the Gateway. The HealthCheck file on the gateway is not going to be modified.

- Check Point does know about it. It's being shared accross the whole company for months.

- Regarding the internet access requirement...
I think it's clearly stated in the full sentence. It is a must IF you want to use Google Drive feature.

@Petar_Markota may be able to explain better if I got something wrong

Please, let us know your thoughts.
Regards,
Henrique

Danny
Champion Champion
Champion

Thanks for the clarifications and corrections.

I'm still curious about the Google Drive thing. Petar confirmed some people (i.e. Check Point SEs/Partners) do upload Check Point CPinfo and other troubleshooting files to Google Drive which is why he added the feature to download files from there.

CPinfo files may contain valid UserCenter account data, customers' entire network topology and security policy information and many other highly confidential data and therefore should never be uploaded to any external resource.

I'd be glad if Check Point's own DiagnosticsView would allow showing Healthcheck reports and CPview database plots for the opened CPinfo/CPview files.

@Mor_Aisenberg , @Anna_Fridman : What about this idea? Obviously Check Point SE's/partner have the need for a simple utility to runs advanced health checks and CPview database plots, let's extend DiagnosticsView for this in order to have one central CP support diagnostics tool.

Petar_Markota
Employee Employee
Employee

GDrive functionality is there since June/July. At that time the tool was accepting only cpview .dat files, or cpview .dats within .gz or .zip, and at that time I saw couple of cases where people for easier sharing are uploading it to other sources. GDrive seemed as one of the nice ways to share stuff, so this is the reason it was added in the first place. CPInfo analysis (healthcheck) has being added in November this year. At this time GDrive functionality was already a part of the tool.

Danny, I fully agree with you that CPInfo isn`t something that should be uploaded anywhere ever, but GDrive is just a functionality which has been added as one of the options for CPInfo analysis too, since it was there for cpview .dats. Healthcheck script is basically the only thing within CPViewer which processes .info files. When I created a GDrive download option, I looked at it from the development perspective and I didn`t think if someone wants or doesn`t want to do this - it was purely to have another way to do it besides the local attachments.

Again, this tool is something that we created for purposes of troubleshooting in our region (because easy and automatized way for getting it visualized nicely did not exist) and after realizing it can really help not only us, but anyone with our gateways, we decided to spread it out.

It would be really awesome if DiagnosticsView would be doing this too. Thumbs up for that!!

Danny
Champion Champion
Champion

So let's sum up what CPViewer currently is:

  • A Linux OS Ubuntu installation within a VMware .ova file (1.1 GB)
    • containing a web server to support uploading for CP diagnostic files
    • containing a Grafana installation to show a database plot of CPview databases within a web browser
    • containing the Gaia Health Check script to provide the HTML result of the script

Correct?

0 Kudos
Petar_Markota
Employee Employee
Employee

Correct, more precisely, It is the Ubuntu (release 18.04) server running the following:


1) Apache v2.4 as a web server (+PHPv7.3, +SQLite3 used in queries);

2) Grafana (v6.5.1) for the data visualization;

3) InfluxDB (v1.7.9) where all the data is stored after extraction;

4) Number of bash scripts with queries which are used for data extraction from .dat files;

5) Python (python3) script using "GDown (v3.8.3)" utility for downloading the files from GDrive;

6) Number of .php scripts which are working in convergance with bash scripts in order to automatize the following:

- Data extraction from .dat files depending on the version (R77.30/R80.10 or R80.20 and above) since two engines differ in queries structure as the versions DBs are different;

- Automatic Grafana datasource provisioning;

- Automatic Grafana dashboard provisioning. Here we created two dashboards R77 and R80 which are used depending on the user`s input at the beginning regarding the versions;

- Automatic Grafana timestamp provisioning;

- Automatic URL provisioning (IP range checks - for cases where the VM is exposed to the access from the internet);

- Running the healthcheck script on cpinfo files and showing a result in .html to the user;

- Uncompressing from .gz, .zip, .info.tar.gz when needed;

7) Crontab lines responsible for automatically deleting all data on weekly basis;

8. Since ubuntu didn`t have preinstalled many small tools which were needed, I installed few along the way, necessary for dev fixes and the tool to run properly (examples open-vm-tools, mlocate, unzip etc. );

9) User interface (some html+css for frontend and JS for input checks on few places) which takes the relevant info from the fields (later used for naming the datasources/cpviews and version pointer);

 

I hope these details are clearing it more up.

/var/www/html contains all the code responsible for the stuff from above.

(1)
Danny
Champion Champion
Champion

Cool, I'm adding this project to my CPX 2020 presentation 'CheckMates - Best of Code Hub Contributions' in Vienna on February 5th, 2020 (14:00 - 14:30).

Petar_Markota
Employee Employee
Employee

Sounds awesome! Thanks a lot @Danny !!

_Val_
Admin
Admin

@Petar_Markota Once more, fantastic job, Petar!

Petar_Markota
Employee Employee
Employee

Thank you @_Val_ !! 

Always happy to contribute to the community 🙂

HeikoAnkenbrand
Champion Champion
Champion

Great job:-)

➜ CCSM Elite, CCME, CCTE
0 Kudos
Anna_Fridman
Employee Alumnus
Employee Alumnus

DiagnosticsView latest version already supports CPView database chart creation (CPView DB Viewer). Please refer to the sk125092 for the detailed description and DiagnosticsView download link (https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...).

Regarding health checks, it is in our future features list.

(1)
Evgeniy_Olkov
Collaborator
Collaborator

Great staff! Thanks!

Could you please share the manual how to change the IP address? I found that it's not easy 🙂 

/etc/network/interfaces doesn't work.

0 Kudos
_Val_
Admin
Admin

Will that help?

https://vitux.com/ubuntu-network-configuration/

Web service most probably should also be restarted though.

JozkoMrkvicka
Mentor
Mentor

Hello,

Why is this thread placed inside the Management section ? Is this tool capable to work only with management data ?

image.png

Why there is no official sk article about this new tool, as it was fully approved/tested/QAed by Check Point employees ?

Why should I download something outside of Check Point servers which was properly tested by authority (Check Point) ?

I am very sorry, but once I first time clicked on this page, I expected that I will be able to download a simple .sh file like is the case for Health Check script.
I simply scrolled it to the end and closed the page, as too complicated and time consuming.

If I may suggest something... Why not put everything as part of SmartConsole (maybe extension) ? Right click on selected gateway --> Monitoring --> Show CPU graphs/ Show Memory graphs etc ? 

Simply combine all features from CPInfo utility  +  CPView UtilityHealth Check script + DiagnosticsView  + CPViewer + other tools mentioned in Support Debug Tools into one universal powerful Check Point proprietary Monitoring tool.

Kind regards,
Jozko Mrkvicka
_Val_
Admin
Admin

@Petar_Markota could you please respond?

From my side:

1. The tools is provided "as is". It is developed by Check Point SEs and not R&D

2. It is based on several open source technologies, and it requires a VM to run on, hence it is virtually impossible to implement it as part of SmartConsole at this point. 

 

However, I like the idea of developing the extension based on a similar idea. Just mind, that effort may take lots of time, considering complexity of the tool. Also, I am not sure R&D has enough time and resources to invest here. It might be, having a VM on a side is the only way. 

Anna_Fridman
Employee Alumnus
Employee Alumnus

Hello,
There are tools that collects data (CPInfo, CPView) and there are tools that analyze it. DiagnosticsView is the official Check Point R&D diagnostics tool for CPInfo and CPView data visualization and analysis. Our intention is to add all features to the DiagnosticsView and have one tool - DiagnosticsView.
For any DiagnosticsView question or feature suggestion, please contact us by sending an email to diagnostics_view_features_box@checkpoint.com

Thanks,
Anna Fridman, Diagnostics team SW developer.

Petar_Markota
Employee Employee
Employee

Hello @JozkoMrkvicka ,

Thanks for the feedback and questions! 

Answers to your questions:

Thread is placed in Visibility&Analytics part since the tool is focused on these.

Like @_Val_  explained, the tool is open-source and provided "as is". It is not an official Check Point tool but rather a open-source community tool available to all who can find it useful. One of the comments above also explains this.

The tool can be set up and ready to go within minutes and is very simple to use. In the tutorial (shown in the video above) - it took 7 mins for importing the VM, logging in, setting up network, attaching cpinfo and cpview files and processing them to the results pages.

The details about the tool in the post are there to guide people and explain the tool as better as possible. 

The idea of the consolidation within SmartConsole which you mentioned is amazing, but this requires valuable resources and a lot of time. On the other hand what Val mentioned - the conversion of all this into something that could be integrated within SC would be much bigger project than the CPViewer itself. 

 

 

MRossi92
Participant

Thanks for the utility.

I have the same scenario but i need to monitoring throughput per interface in 64000 Chasis.

How can i do that? I find some OID but those are incorrect.

Thanks.

 

0 Kudos
Petar_Markota
Employee Employee
Employee

There is a graph through which you can see RX&TX for each interface called "Interface throughput". Example ss attached.

MRossi92
Participant

Thanks for the graph, we can try to do something like this.

Can you share me the "root" "OID" to get this graph? And also we need the query to execute in Grafana

Remember we have a 64000 Chasis with a lot a lot of vlan in two bonds.

Thanks !!!!

 

0 Kudos
Petar_Markota
Employee Employee
Employee

root?

Within the platform, we are not getting any of the info based on OIDs.

We are using the following query for extraction of interface name, RX and TX throughput from cpview.dat file:

sqlite3 -noheader -csv $DB "SELECT if_name,if_tx_bits_throughput,if_tx_packets_throughput,Timestamp FROM UM_STAT_UM_HW_UM_IF_TX_STATISTICS_TABLE WHERE if_name NOT LIKE 'TOTAL'"

Results example which query above returns attached.

In Grafana you are able to see the query if you go to "edit" button on "Interfaces throughput" (while having a graph populated) -> query ss attached.

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events