FilterConfiguration.xml gets overwritten at restart of cp_log_export

[Obiwan1968]

Hi There

Using R80.30 Take 191
I have seen the example in sk122323 about the FilterConfiguration.xml.

Using the example below by editing $EXPORTERDIR/targets/<target-name>/conf/FilterConfiguration.xml

<filters>
        <filterGroup operator="and">
                <field name="action" operator="and">
                </field>
                <field name="origin" operator="and">
                </field>
                <field name="product" operator="or">
                        <value operation="eq">SmartDefense</value>
                        <value operation="eq">Threat Emulation</value>
                </field>
                <field name="severity" operator="or">
                        <value operation="eq">3</value>
                        <value operation="eq">4</value>
                </field>
        </filterGroup>
</filters>

But when I restart the cp_log_export service, all changes are gone and the file "FilterConfiguration.xml" is "empty" again as before? Anymore any idea why this file gets "reset" be the restart of the service?

[Dror_Aharony]

It means the file wasn't legal, therefore getting overwritten with the default file on restart.

Try quoting the Threat Emulation: "Threat Emulation".

*Copy/Save text of file in any-case before restarting for easier repeat editing attempts.

Let us know if it works.

[Obiwan1968]

Is a "cp_log_export reconf" necessary or only doing a "cp_log_export restart"?

With "reconf" the file gets default again, with "restart" the FilterConfiguration.xml keeps its settings, but they do not work, nothing appears. I tried with 


<filterGroup operator="and">
<field name="action" operator="and">
</field>
<field name="origin" operator="and">
</field>
<field name="product" operator="and">
<value operation="eq">IPS</value>
</field>
</filterGroup>
</filters>

Without the IPS line I get logs with product IPS/Firewall etc, with the line nothing at all

[Dror_Aharony]

no, no. only restart. 

reconf is only meant to run once post upgrade to update the log-exporters to updated version (including base files)

Do not repeat it afterwards, or it will override your files, as you've already seen.

Great, then file is good:)

Not IPS, as it uses the raw/older name (SmartDefense).
see Log Fields Description sk144192 > https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

IPS=SmartDefense.

and afterwards add "Threat Emulation" if you like.

You can also use the suggested Threat group (filter-blade-in "TP") for all threat logs.

see log-exporter sk122323 for more examples > https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

Hope that solves it.

[Obiwan1968]

Now it looks good, I get traffic this way

[Obiwan1968]

Can I do something like this? Does not work. The idea is having the SmartDefense with certain severity and confidance AND the AUDIT logs

When I kill the audit log, the SmartDefense in itselfe works, but I am not sure if I can use cases this way? or what the right way is?

<filters>
        <filterGroup operator="or">
                <filterGroup operator="and">
                        <field name="product" operator="or">
                                <value operation="eq">SmartDefense</value>
                                <value operation="eq">TP</value>
                        </field>
                        <field name="severity" operator="or">
                                <value operation="eq">2</value>
                                <value operation="eq">3</value>
                                <value operation="eq">4</value>
                        </field>
                        <field name="confidence_level" operator="or">
                                <value operation="eq">3</value>
                                <value operation="eq">4</value>
                                <value operation="eq">5</value>
                        </field>
                </filterGroup>
                <filterGroup operator="and">
                        <field name="product" operator="or">
                                <value operation="eq">SmartConsole</value>
                        </field>
                </filterGroup>
        </filterGroup>
</filters>
[

[Dror_Aharony]

There is no value TP, it's a set group filter for the filter-blade-in that translates to all Threat blades:
SmartDefense, "Threat Emulation" & more...

Sorry, I think you did the Audit, right.

Let's see & I'm also checking here.

[Obiwan1968]

I am not sure is this structure with "filterGroup" is correct? Is there any big example how to do this correctly?

[Dror_Aharony]

I understand what you mean, but you configured it properly.  I verified it.

Top filterGroup "or" between the groups of FW-filtered & Audit.

Another filterGroup "and" between the fields (IPS, Severity,Confidence).

Another filterGroup "and" between the Audit field (even if it's just one).

filter of fields "or" between actual values in the same field (even if it's just one).

Are you sure you don't see any Audit logs being exported?

Anybody, @PhoneBoy  @Shay_Hibah?

[Obiwan1968]

Ok I delete the "TP" line after "SmartDefense"

When I run this, I suddenly get plenty of log entries from the "product=Firewall" which I do not want, no idea why? I just need the SmartDefense and the Audit Logs, but no firewall logs

<filters>
        <filterGroup operator="or">
                <filterGroup operator="and">
                        <field name="product" operator="or">
                                <value operation="eq">SmartDefense</value>
                        </field>
                        <field name="severity" operator="or">
                                <value operation="eq">2</value>
                                <value operation="eq">3</value>
                                <value operation="eq">4</value>
                        </field>
                        <field name="confidence_level" operator="or">
                                <value operation="eq">3</value>
                                <value operation="eq">4</value>
                                <value operation="eq">5</value>
                        </field>
                </filterGroup>
                <filterGroup operator="and">
                        <field name="product" operator="or">
                                <value operation="eq">SmartConsole</value>
                        </field>
                </filterGroup>
        </filterGroup>
</filters>

[Dror_Aharony]

These may be IPS as FW logs, we have a few of these.

You can use the additional filter-out of FW connections.

Change to true in targetConfiguration.xml.

<filter filter_out_by_connection="true">

Any updates on the Audit logs?

[Obiwan1968]

With this minimal configuration it works, but here I have also "informal","low" and "low-medium" SmartDefense" events listed

<filters>
        <filterGroup operator="and">
                <field name="product" operator="or">
                        <value operation="eq">SmartConsole</value>
                        <value operation="eq">SmartDefense</value>
                </field>
        </filterGroup>
</filters>

[Shay_Hibah]

Hey @Obiwan1968 

How are you?

Filter configuration supports (for now) only one filterGroup.

This file should be saved during restart and no additional command is needed to keep it with the new configuration.

Can you please let me know what exactly you wish to achieve and I will try to help?

Thanks,

Shay

[Obiwan1968]

Hi Shay

I wanna do two things:

1) all audit logs
2) all attacks with severity (2/3/4) and confidence_level (3/4/5)

[Obiwan1968]

Just comes to my mind, I could of course create two instances of cp_log_export services instead of putting all the settings into one?

[Dror_Aharony]

Exactly:)

[Paul_Hagyard]

Hi,

While I love the fact that it can be made to work with two instances, this presumably means there are now two processes parsing all of the log messages, rather than one with a more complex filter. I'm unsure of the performance implications of the two approaches, but I would have thought one process could be more efficient in terms of IO and performance.

It's a couple of years since this post, has there been any work to support more complex filters, through multiple filterGroup entries or some other mechanism?

Cheers,

Paul

[PhoneBoy]

As far as I know, this hasn't changed and would require an RFE.
Having said that, running multiple instances is expected in some configurations independent of this use case.
At least on the community side, I haven't seen any specific complaints related to this.

[Shay_Hibah]

You are right.
In case you want to export all audit logs and in addition, export all security logs with severity 2+ and confidence 3+, it is possible to create 2 exporters:
1. one of the exporters will export only audit logs (can be configured in targetConfiguration.xml file).
2. The second exporter will export all security logs with filtering configuration as mentioned above.

Do you need any assistance to do this?

[Obiwan1968]

No, I am fine and everything works now as expected 🙂

Thanks for your support!

[Cesar_Almada]

hello

could you share the xml, please

i need the same filter

thanks

Cesar A.

[Dror_Aharony]

One log-exporter filter for the IPS severity/confidence logs, like this:

<filterGroup operator="and">
                        <field name="product" operator="or">
                                <value operation="eq">SmartDefense</value>
                        </field>
                        <field name="severity" operator="or">
                                <value operation="eq">2</value>
                                <value operation="eq">3</value>
                                <value operation="eq">4</value>
                        </field>
                        <field name="confidence_level" operator="or">
                                <value operation="eq">3</value>
                                <value operation="eq">4</value>
                                <value operation="eq">5</value>
                        </field>
                </filterGroup>

Add Filters to start & to close as well.

<filters>

/filters>

2nd for audit only:

Configure your targetConfiguration.xml to audit only, without any filters, like this:
<log_types>audit</log_types><!--all[default]|log|audit/-->

Hope that's clear enough (as I used the current pictures).

[omidrajaee]

Hello, Can you please let me know how did you fix the problem? I used the same template as yours but I am not getting everything!

[omidrajaee]

Hi, 

Only Smart Defense and Threat Emulation are enough how I can find out the other services or running on my CP? DO I need to use Confidence level this is what I've used along with severity but I missed all unknow logs I had before xml editing:

                <field name="product" operator="or">

                        <value operation="eq">SmartDefense</value>

                        </field>

                        <field name="severity" operator="or">

                                <value operation="eq">2</value>

                                <value operation="eq">3</value>

                                <value operation="eq">4</value>

                        </field>

                        <field name="confidence_level" operator="or">

                                <value operation="eq">3</value>

                                <value operation="eq">4</value>

                                <value operation="eq">5</value>

                </field>

[PhoneBoy]

From the gateway itself? enabled_blades
From the management server, refer to the relevant gateway object.

[Dror_Aharony]

If I understood you correctly, you wanted the confidence_level unknown to be exported as well?
Add a line for 0 (=Unknown) as well.

<field name="confidence_level" operator="or">

                                <value operation="eq">0</value>

[omidrajaee]

Hello, 

It did not help! also I am only getting the SmartDefense event nothing else. Do you know how I can get Firewall, VPN, URL Filtering and etc? Before filtered out I got some unknown severity logs and they are very useful but now I don't have them anymore.