Gateway RADIUS login suddenly "unable to get user permissions"

[basinUnaltered]

I've been gradually enabling RADIUS authentication on our gateway fleet but today I noticed that when trying to sign in via SSH I'm receiving an error when using my AD credentials. The last time I tried was on Friday and it was working then, and so far I haven't found any indications of someone else on my team making changes.

The error is:

/bin/shell_wrapper.sh: line 7: /dev/null: Permission denied
Unable to get user permissions.
Connection to 172.16.1.1 closed.

aaa configuration is:

add aaa radius-servers priority 2 host 192.168.12.3 port 1812 secret ***** timeout 3
set aaa radius-servers NAS-IP 172.16.1.1
set aaa radius-servers default-shell /bin/bash
set aaa radius-servers super-user-uid 0

Any ideas what might have caused this sudden issue?

[the_rock]

Can you send the content of /etc/ssh/sshd_config file? 

Andy

[basinUnaltered]

Thanks Andy, here it is. (ip addresses have been changed)

#  This file was AUTOMATICALLY GENERATED
#  Generated by /bin/sshd_template_xlate on Mon Apr  7 14:48:32 2025
#
#  DO NOT EDIT
#
#       $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

# If you want to change the port on a SELinux system, you have to tell
# SELinux about this change.
# semanage port -a -t ssh_port_t -p tcp #PORTNUMBER
#
#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

#RekeyLimit default none

# System-wide Crypto policy:
# This system is following system-wide crypto policy. The changes to
# effect here. They will be overridden by command-line options passed on
# the server start up.
# To opt out, uncomment a line with redefinition of  CRYPTO_POLICY=
# variable in  /etc/sysconfig/sshd  to overwrite the policy.
# For more information, see manual page for update-crypto-policies(8).

# Logging
#SyslogFacility AUTH
SyslogFacility AUTHPRIV
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#PubkeyAuthentication yes

# The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
# but this is overridden so installations will only check .ssh/authorized_keys
AuthorizedKeysFile      .ssh/authorized_keys

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
IgnoreUserKnownHosts yes
# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no
PasswordAuthentication no

# Change to no to disable s/key passwords
#ChallengeResponseAuthentication yes
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
#KerberosUseKuserok yes

# GSSAPI options
##GSSAPIAuthentication yes
#GSSAPICleanupCredentials no
##GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no
#GSSAPIEnablek5users no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
# WARNING: 'UsePAM no' is not supported in Fedora and may cause several
# problems.
UsePAM yes

#AllowAgentForwarding yes
AllowTcpForwarding no
#GatewayPorts no
X11Forwarding no
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes

# It is recommended to use pam_motd in /etc/pam.d/sshd instead of PrintMotd,
# as it is more configurable and versatile than the built-in version.
PrintMotd yes

#PrintLastLog yes
TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveCountMax 3
#ShowPatchLevel no
#PidFile /var/run/sshd.pid
MaxStartups 10
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
Banner /etc/issue

# Accept locale-related environment variables
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS

# override default of no subsystems
Subsystem       sftp    /usr/libexec/openssh/sftp-server

DenyUsers root shutdown halt nobody ntp pcap rpm
# ckp - CR00596011
#AllowGroups root

# Example of overriding settings on a per-user basis
#Match User anoncvs
#       X11Forwarding no
#       AllowTcpForwarding no
#       PermitTTY no
#       ForceCommand cvs server

Match address 192.168.1.0/24
        PasswordAuthentication yes
Match all

Ciphers chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
Macs umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
UseDNS no
PermitRootLogin yes
ClientAliveInterval 0

[the_rock]

Looks right to me. See if something is different in file below (I got this from my lab).

Andy

[Expert@CP-MANAGEMENT:0]# more /bin/shell_wrapper.sh
#!/bin/sh -

SHELL=`cat /etc/radius_def_shell`

# Validation to check if user permitted to log-in with clish permissions
if [ "$SHELL" != "/etc/cli.sh" ] && [ "$SHELL" != "/etc/gcli.sh" ]; then
$SHELL -c "clish -c ver" > /dev/null 2>&1
if [ $? -ne 0 ]; then
echo "Unable to get user permissions."
exit 1
fi
fi

if [ -n "`id -G | grep 97`" ]
then
export PATH=$PATH:/usr/sbin:/usr/local/sbin:/usr/local/bin:/sbin:/home/admin
/bin
sudo -- $SHELL "$@"
else
exec $SHELL "$@"
fi

exit 0

[basinUnaltered]

Ours look the same

Weirdly, I just rebooted my lab unit and RADIUS works again on it. Comparing it to a prod device where I'm still getting the error and everything looks the same except for permissions on /dev/null:

[Expert@FWLAB:0]# ls -la /dev/null
crw-rw-rw- 1 admin root 1, 3 Apr 23 11:12 /dev/null


[Expert@FWPROD:0]# ls -la /dev/null
crwxr-xr-x 1 admin root 1, 3 Nov 23 09:06 /dev/null

Doing some reading it sounds like /dev/null permissions should always be 666. Do you know of anything that would cause the gateway to change /dev/null permissions on its own?

[the_rock]

Never seen it myself. To be sure, I would just run chattr +i on that file and call it a day. Before that, maybe do chmod 777 

Andy

[basinUnaltered · Accepted solution]

It just occurred to me that I did create a new user in Smartconsole around that time, but it didn't register to me at first since that change was on the management server and not installed on the gateways. My best guess is that something about that then changed the /dev/null file on some of the gateways, but not sure how/why. Guess I'll just update the permissions and move on.

Thanks Andy

[the_rock]

Sounds good! Below may not work, but not a big iissue 

Andy

[Expert@CP-MANAGEMENT:0]# chattr +i /dev/null
chattr: Operation not supported while reading flags on /dev/null
[Expert@CP-MANAGEMENT:0]#

[the_rock]

from my lab:

[Expert@CP-MANAGEMENT:0]# ls -lh /dev/null
crwxr-xr-x 1 admin root 1, 3 Apr 20 09:21 /dev/null
[Expert@CP-MANAGEMENT:0]#

[basinUnaltered]

Strange; do you have RADIUS auth configured for SSH login on your lab?

[Bob_Zimmerman]

I get this all the time. On my end, it seems to be a problem with RBA roles on the firewall, which seems to be triggered somehow by something in Cisco ISE (the RADIUS server we're using). We haven't yet tracked down what is going on, as we only really notice it when it happens during an outage.

[travelmaniac]

The same problem started occuring in our Environment a few days ago.
We get the same error on various Checkpoint Instances (MDS, MLM, Firewalls) since a few days.
(not all at the same time, but every few days the next.)

Also, in our case the permissions of /dev/null have suddenly changed to:
crwxr-xr-x 1 admin root 1, 3 Mar 17 18:39 /dev/null

So far this behaviour is only at 81.20 Take 98.
(older versions seemed not to be affected)

[the_rock]

What version?

[travelmaniac]

The unaffected versions are mostly 81.20 take 89.

[Peter-Kyn]

I can confirm: we updated end of March our Management to  81.20 Take 98 and via onetime script in SmartDashboard I can see now that permission of /dev/null has changed to:

crwxr-xr-x 1 admin root 1, 3 Mar 28 08:43 /dev/null

but this has happened somewhere last week, as I could still logon on Monday last week but not now...

Will try to change it back via One Time Script, let's see how long it will be unchanged...

[the_rock]

The only problem could be if chattr +i cant be done on it, it may not keep the changes after the reboot : - (

Andy

[-K-]

No new users added, the strange thing is with VSX platform it works fine but for physical firewall i see this same issue happening and thats on all multiple firewalls. 

any solution on this issue ?

[the_rock]

I would open TAC case and see what they say. When I had this issue in the past, I always found reboot would fix it.

Andy

[IDoCare2022]

We have a client with dozens of r81.10 HFA 170 gateways impacted by this as well. All of a sudden those /dev/null file permissions changed. ( in our case, also an ISE client ) No radius logins any more, total panic, only local admins can login. Really annoying, TAC ticket is logged because we need the root cause, it could happen again any time soon, who knows...

Don't know what we should do in the mean time : chattr +i /dev/null ? crontab the proper rights every hour ? ...

[the_rock]

Yea, you can try last thing you said

[basinUnaltered]

Like Andy noted and reboot seemed to fix it in my lab. Most of my sites run 24/7 so I can't reboot prod without scheduling a maintenance window but applying a one-time script of 'chmod 766 /dev/null' to all my gateways through SmartConsole did the trick. If clish is your default shell that might be a bit more of a pain, not sure how to run bash commands direct from there.

[the_rock]

Very odd indeed.

[travelmaniac]

yes, a reboot fixes it (at least temporarely).
I think this is, because  the /dev/ filesystem gets recreated during the reboot.

[YounesIP10]

I became aware of this issue this week from some friends and CheckPoint confirmed this is a bug that their RND are working on, and this is what we were able to understand:

• Version: R81.20 Jumbo Hotfix Take 98
• If you have the following setup on your AAA config "set aaa radius-servers default-shell /bin/bash" you will not be able to get to connect because of the following
• Trigger: after a reboot the "/dev/null​" will have the privilege move from "crw-rw-rw-" to "crwxr-xr-x". This would prevent authentication even with correct because AAA config because the Write mode required for "/dev/null​"
• Solution:

1. Access the Firewall using Web Interface and create local user that can land on Expert mode directly "bin/bash"
2. SSH to the Firewall using this local account and readjust the permission using the following: "chmod 666 /dev/null"
3. And you should be good

If you have many Firewalls that are managed via SmartConsole, you can do the same from the SmartConsole scripting functionality to do it massively

Hope this can help

[the_rock]

Great suggestion!

[Nik_Bloemers]

The jumbo take information seems to be incorrect. We have multiple clusters this issue started occuring recently, but these are all still running JHF 92.

The permissions fix does work though, which is easy enough to roll out from SmartConsole with a one-time script.

[iblake1]

Having the same issue. I noticed that the permissions changed on the /dev/null file, and right around the same time in the messages file is an indication of an update to CPotelcol.

Run the command "stat /dev/null" 

Look at the line that says "Change:  <date>"

Then look in your /var/log/messages at the same time. You should see that CPotelcol was updated.

[Expert@a:0]# stat /dev/null
File: '/dev/null'
Size: 0 Blocks: 0 IO Block: 4096 character special file
Device: 12h/18d Inode: 1173 Links: 1 Device type: 1,3
Access: (0755/crwxr-xr-x) Uid: ( 0/ admin) Gid: ( 0/ root)
Access: 2025-04-14 10:28:24.503999999 -0400
Modify: 2025-04-14 10:28:24.503999999 -0400
Change: 2025-04-23 12:07:03.440741432 -0400

[Expert@firewalla:0]# grep CPotel /var/log/messages*

messages.1:Apr 23 12:07:25 2025 firewalla AutoUpdater[30809]: Installation succeeded for component CPotelcol

[the_rock]

Can you try do chattr +i on it and see if it works?

Andy

[iblake1]

I'm sure that would fix it. But obviously this is an issue caused by something external. All my gateways were fine, and slowly as they received this update I lost the ability to log in with anything other than an admin account.  And I would prefer to not have to log into over 50 gateways and fix each one. 
I'm hoping a new update can be pushed to resolve the issue.

[the_rock]

Cause I tried that in the lab, but command failed...

Andy