Best Practices for Identity Collector Architecture

[Josh_Wilson]

Is there a "best practices" doc available that gives coverage of proper IDC architecture, specifically in VSX environment with multiple VS running IDA? Should IDC agents be configured with each IDA enabled VS as a gateway (IDC agent side)? Should only VS0 run IDA and share out the database to each VS? I'm having a difficult time finding the best way to implement this IDC on VSX in regard to reliability first, redundancy second, and performance third.

Thanks,

Josh

[Quinn_Yost]

I've been waiting for an IDC best practices document since it was in EA.   Someday...

[PhoneBoy]

You'd think Identity Collector would be mentioned here: Best Practices - Identity Awareness Large Scale Deployment 

I submitted some feedback in the above SK.

Hopefully this is something we can address in the near future.

[Matt_Taber]

6 months later, no sign of IDC in the SK: Best Practices - Identity Awareness Large Scale Deployment

Do we have any thoughts on when the SK may be updated?

Thanks

[Tzvi_Katz]

Hello, 

sk88520 covers Identity Collector as well. Essentially IDC should be treated as any other identity source, one (or more) PDP should be defined and the rest of the enforcement points should get the information using identity sharing. 

[Scott_Chambers]

Is there any limitations with the # of IA collectors and sharing across gateways?

We have a rollout at my company where 9 of the 10 clusters have local AD servers.    Our windows team had concerns with AD load and hence why we went with IA collectors verse AD Query.

In order to be as redundant as possible, we attempted the following:

• Each of the 9 sites have 2 local IA collectors connected to its local AD servers only 
• Each of the 9 sites shares (or attempts to) with the other locations
• The 10th site has no AD servers locally so it connects to one of our datacenter's IA collectors (both IA servers).
• The 10th site accepts shares from other gateways but does not share themselves
  • Since its collecting from the same IA as the DataCenter GW, that DC GW is already sharing.

What we are getting is a whole mix of users that seem to collect fine locally but simply do not share across all other GWs.

Notes:

2 locations run clustered 13500 appliances

6 locations are 4800 clustered appliances

1 location (the one with no AD servers) is a 5800 cluster.

When looking that the "best practices' docs for IA, it doesn't seem clear to me on what method would be best for reduced load, efficiency and most importantly....redundancy.

Questions:

• Has anyone had similar issues with this type of rollout?  
  • What did you do to get it to work?
• What are the # of redundant IA collectors you can deploy to one gateway?
  • Can you do more than 2?
• What is less on resource load and bandwidth:   
  • Deploying 2 (or more )IA collectors in 2 different DCs (redundancy) that collect logs from EVERY AD server across the enterprise; have every GW collect from these IA servers and do not share identities
  • Deploying 2 (or more )IA collectors in 2 different DCs (redundancy) that collect logs from EVERY AD server across the enterprise; have only the DC GWs connect to the IA collectors and share identities with all other GWs
    
    • In this event, would I need to deny sharing between the DCs since they are collecting the same logs?
    • Is there any issue with the remote GWs since they would see the same sharing info from both DC GWs?

[Scott_Chambers]

Additional info:
Management server: R80.20 JHF take 47
GW: All are R80.10 JHF take 189

[Vladimir]

I too would like to get better understanding of IA in general and IDC use cases in particular.

It does not look like we can use IDC for Identity Logging, which still relies on the high privilege account with AD query.

Would be nice to remove AD Account Units from the picture and rely on IDCs for that as well. 

[Norbert_Bohusch]

There are 2 problems with Identity Sharing:

1. Identity Sharing via VPN can cause IPSec Replay Attacks by clusters standby member

2. Identity Sharing between gateways managed by different domains of a MDM is a pain

That's why if only IDC is involved, I tend to configure each gateway to receive the events individually.

But with other identity sources like TSagent, RemoteAccess this is not possible, as they are not shared then.

[Scott_Chambers]

Norbert,

Luckly, neither of the 2 problems you have defined would affect us. I ended up changing the topology last night for 2 clusters (2 separate data centers) connected to the same IDCs (one in each DC) and then sharing with the other gateways.

Looks ok so far but will need to test further as old user sessions start to age out.

I was thinking of having each GW just all connect to the same 2 IDCs but I know that a terminal agent for citrix is in our future and i might have to use the user agents for our MAC OS clients here.

Captive portal is something we wanted to avoid since we already have one for forcepoint in play today.

[Scott_Chambers]

Quick update:

The IA collectors at 2 DCs and then sharing to other gateways has been working great!    We gotten a few access roles deployed and so far its working as we imagined.

For the MAC issue, this is really due to MAC OS.     With any MAC OSX joined to AD, the MAC is not 'site aware' and therefore we were getting auths sent to DCs that we were 1) not monitoring with the collector and 2) really didn't want them going randomly cross site.

The temp fix was to set a prefered AD server in the Active Directory config on the MAC.   That has consistantly gotten the login but have to sometimes 'force' the update with "kinit" via terminal.

Long term (outside of using the machine agent from Checkpoint), we have been testing NOMAD with the MACs.   NOMAD is site aware and our initial testing gets the MAC as close to windows in regards to AD.

One item did come up that caught us offguard:   2 of DCs we are collecting from stopped connecting.   Its for a sister company of ours and think something changes on a config or patch.    Regardless, my question is:   how do you monitor this type of event on the Identity Collector itself?  

I've seen post on custom querries to the gateways to detect when they stop getting feeds from the collector itself.   I've also seen the updated SK (sk108235) on the monitoring capabilities via SNMP to R80.20 gateways for the collector status.

I simply can't find in any searches or admin guides on how to monitor issues on the IA collector itself in regards to its "identity sources" status. 

Does anyone know how to detect such an event?

Lastly, during my search, I see that there is a newer collector out (sk134312).   We are currently running 80.72.0000  and I'd like to plan to upgrade it......but can't seem to find any docs on how to upgrade 🙂     Plenty on install but nothing on upgrading a existing collector.

• Do you export the config, delete the agent and reinstall?
  • Any changes needed to resynch with the gateways, its identity sources, etc?
• Does it support a direct upgrade when running the new MSI package?
  • Does the configs stay in a 'in place' upgrade? 

[Scott_Chambers]

Anyone have any advise or reference documentation on how to upgrade an existing IA Collector MSI?

I still can't find any notes on checkmates or within support center.    

Thanks in advance 🙂 

[Scott_Chambers]

Anyone have info on upgrading the Identity Collector agents?

[Royi_Priov]

Hi @Scott_Chambers ,

Identity Collector has a monitoring feature, which reflects the "Identity Sources" tab on the IDC to the GW.

As you have mentioned, it will allow you to monitor changes with sources which are disconnected.

There are 3 main ways to get this data:

• "cpstat identityServer -f idc" command on the GW
• add SNMP query for the relevant OIDs (can be found in  $FWDIR/conf/identity_server.cps).
• in R80.30, we have also added "pdp idc status" command to see the output.

all is mentioned in sk108235, including the steps needed to enable this feature.

As for installation, you can run in-place installation, which will overwrite the application files while saving your config.

I do suggest saving an "export" via the IDC before running the upgrade, just to be on the safe side.

If any other questions are raised, please tag me and I will take a look.

Thanks,

Royi Priov - Identity Awareness R&D.

[Scott_Chambers]

@Royi_Priov 

Thank you for the response.    Glad to hear that the upgrade of the agents are that simple 🙂

I'll be upgrading ours tomorrow and will provide feedback.

In regards to the monitoring, I see the SK regarding the GW to IA collector via CLI or SNMP (R80.20 GW and up).   

One thing I haven't been able to find is additional monitoring on the IA collector itself (or its server its installed on).   

We had a situation a few weeks back with a partner company and they 'accidentally' set our AD service account for the collector to expire.  

Outside of event monitoring of this account expiring in SmartConsole, is there a way to alert on the IA collector itself when it can no longer fetch accounts/security events from a select AD server?    The next time might not be an account expire but a permission or server related that would not generate such log.  

We have 32 connected to our IA collectors and this issue affected access to 5 of them.   I would be nice to have alerting from the collector end when such events occur.

While the enhanced monitoring you've mentioned can catch when the IA collector disconnects or has not sent events in the last 'x' hours, an alert from the collector against the AD servers it fetches security events from would be beneficial 🙂

Its very rare for us not to see an event from a AD server so an alert where we don't get an event from a select AD server in 1 hour (for example) would greatly help as we scale out IA across our enterprise.  

[Norbert_Bohusch]

The monitoring on the gateway gets information of events sent for each domain controller. So you can monitor the events and alert if they drop near zero!

[Scott_Chambers]

@Norbert_Bohusch 

Is that true in a IA collector configuration only?

There is no AD Query, API, Terminal/User agents, etc.....just IA collectors and gateway sharing 🙂

I have collectors connected to 2 gateways and the rest is pure IA sharing from those 2.    

My understanding is the IA Collector gets the security events from the AD servers and its that connection I would like to monitor.    If there are events that I would see in SmartLog/Event in this type of setup, can you provide a few examples?  

If its just building out some alert rules for logs that are currently (or can be) sent, I can do that 🙂 

[Norbert_Bohusch]

This monitoring is only for IDC (identity collector), yes.

So if you have a look at the gateway with the following (output from my lab):
[Expert@nb-ckp-gw1:0]# cpstat identityServer -f idc

Identity Collector Sources
--------------------------------------------------------------------------------------------
|Type|Name |Host |Status |IDC IP |Events Recieved|Total Events|
--------------------------------------------------------------------------------------------
|AD |nb-win-ad.nb.lab.local|10.2.231.60|Connected|10.2.231.61| 10| 61351|
--------------------------------------------------------------------------------------------

You see one line for each domain controller and it shows "Events Received".

This is exact the same information you can query via SNMP:

[Expert@nb-ckp-gw1:0]# snmpwalk -v 2c -c public 127.0.0.1 1.3.6.1.4.1.2620.1.38
SNMPv2-SMI::enterprises.2620.1.38.1.0 = STRING: "Identity Awareness"
SNMPv2-SMI::enterprises.2620.1.38.2.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.3.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.4.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.5.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.6.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.7.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.8.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.9.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.10.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.11.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.12.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.13.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.14.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.15.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.16.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.17.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.18.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.19.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.20.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.21.0 = Gauge32: 11
SNMPv2-SMI::enterprises.2620.1.38.22.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.23.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.24.1.1.1.0 = Gauge32: 1
SNMPv2-SMI::enterprises.2620.1.38.24.1.2.1.0 = STRING: "nb-ckp-gw"
SNMPv2-SMI::enterprises.2620.1.38.24.1.3.1.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.24.1.4.1.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.24.1.5.1.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.24.1.6.1.0 = Gauge32: 1
SNMPv2-SMI::enterprises.2620.1.38.26.0 = Gauge32: 3
SNMPv2-SMI::enterprises.2620.1.38.27.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.28.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.29.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.30.0 = Gauge32: 3
SNMPv2-SMI::enterprises.2620.1.38.31.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.32.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.33.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.34.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.35.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.36.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.37.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.38.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.39.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.40.0 = Gauge32: 3
SNMPv2-SMI::enterprises.2620.1.38.41.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.42.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.43.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.44.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.45.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.46.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.47.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.48.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.49.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.50.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.51.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.52.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.53.1.1.1.0 = Gauge32: 1
SNMPv2-SMI::enterprises.2620.1.38.53.1.2.1.0 = STRING: "AD"
SNMPv2-SMI::enterprises.2620.1.38.53.1.3.1.0 = STRING: "nb-win-ad.nb.lab.local"
SNMPv2-SMI::enterprises.2620.1.38.53.1.4.1.0 = STRING: "10.2.231.60"
SNMPv2-SMI::enterprises.2620.1.38.53.1.5.1.0 = Gauge32: 3
SNMPv2-SMI::enterprises.2620.1.38.53.1.6.1.0 = STRING: "Connected"
SNMPv2-SMI::enterprises.2620.1.38.53.1.7.1.0 = STRING: "10.2.231.61"
SNMPv2-SMI::enterprises.2620.1.38.53.1.8.1.0 = Gauge32: 61366
SNMPv2-SMI::enterprises.2620.1.38.101.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2620.1.38.102.0 = STRING: "OK"
SNMPv2-SMI::enterprises.2620.1.38.103.0 = STRING: "OK"

So if you query the value of "events received and see an abnormal low value for one or more AD servers, your monitoring can trigger an alert.

Then you know you don't receive events from a specific AD server or set of AD servers and can investigate, what exactly is broken.

[Juraj_Skalny]

Hello Norbert,

what is a little bit downside is that currently PDP gateway is able to monitor only one IDC.
we have 2 IDC connected to PDP for redundancy and only one is visible when "cpstat identityServer -f idc" or SNMP query is sent to the gateway.
when "pdp con idc" is run I can see two IDC with valid shared secret status delivering events to the gateway

or am I doing something wrong here please?

Thanks,

Juraj

[Juraj_Skalny]

found it!

there must be MonitoringEnabled registry created on both IDC

Then they bot appear when "pstat identityServer -f idc"  is runor SNMP query is sent to PDP.

registry print screen should be in attachment