Machines in EPG (Cisco ACI) are rebooted but not updated on Gateway

[RayP]

Envirornment:

R81.20 JHF T92 Cluster HA with 2 members.
Identity Awareness blade with CloudGourd Controller (Cisco ACI).

When servers are rebooted within an EPG on Cisco ACI, traffic is being dropped by the cleanup access rule.

The reason is that it doesn't hit the accessrules where EPG objects are used.

the command #pep show user query cid <ip address of server> doesn't show me the Identity Role after a reboot.

Before the reboot:

[Expert@gateway:0]# pep show user query cid 1.2.3.4
Command: root->show->user->query

PDP: <127.0.0.1, 00000000>; UID: <ee315f18>
==================================================
Client ID : <1.2.3.4, 00000000>
Authentication Key : <Unavailable>
Brute force counter: 0
Username :
Log Username :
Machine name : epg-workspace
User groups : <Unavailable>
Machine groups : <Unavailable>
Compliance : <Unavailable>
Identity Role : <epg-workspace>
Time to live : 604830
Cached time : 86400
TTL counter : 0
Time left : 587914
Client type : Identity Awareness API
Last update time : Thu May 1 10:22:02 2025

Backup Pdps :

After the reboot of server 1.2.3.4 within the EPG group "epg-workspace" on Cisco ACI

[Expert@gateway:0]# pep show user query cid 1.2.3.4
Command: root->show->user->query

PDP: <127.0.0.1, 00000000>; UID: <ee315f18>
==================================================
Client ID : <1.2.3.4, 00000000>
Authentication Key : <Unavailable>
Brute force counter: 0
Username :
Log Username :
Machine name : servername1
User groups : <Unavailable>
Machine groups : <Unavailable>
Compliance : <Unavailable>
Identity Role : <>
Time to live : 43230
Cached time : 86400
TTL counter : 43170
Time left : 42650
Client type : Identity Collector
Last update time : Thu May 1 14:44:49 2025

Backup Pdps :

In the output above, the IP adddress has been masked by fictitious ip addresses, machine names and identity roles.

Within the SmartConsole -> Data Center Objects the EPG object contains the ip address of the rebooted servers but it has a timestamp behind "Updated on Data Center" of this morning a couple of hours before the reboots.

Why is the gateway not automatically updated with the identity role of the epg-workspace?

Update:

It looks like after a policy install on the gateways, the #pep show user query cid <ip address> is changed and it's working again.

The client type id has also been changed from "Identity Collectors" to "Identity Awareness API".

PDP: <127.0.0.1, 00000000>; UID: <ee315f18>
==================================================
Client ID : <1,2,3,4, 00000000>
Authentication Key : <Unavailable>
Brute force counter: 0
Username :
Log Username :
Machine name : epg-workspace
User groups : <Unavailable>
Machine groups : <Unavailable>
Compliance : <Unavailable>
Identity Role : <epg-workspace>
Time to live : 604830
Cached time : 86400
TTL counter : 0
Time left : 604813
Client type : Identity Awareness API
Last update time : Thu May 1 15:44:52 2025

Backup Pdps :

Why isn't this done automatically without a policy installation ?

[AaronCP]

Hey @RayP,

Have you checked the cloud_proxy.elg log for any relevant error messages? If you have an SMS, the file should be located in $FWDIR/log. On an MDS it's located in $MDS_FWDIR/log. Might be worth a nose in there. I've experienced similar issues before where the EPG traffic wasn't matching the Cisco ACI object in policy, even though the Cisco ACI object showed that trust was established. After checking the cloud_proxy.elg file, it showed that our MDS had lost trust with the APIC, but it wasn't obvious when checking the dashboard object.