Tip: How to deal with RAD errors

[Lesley]

RAD errors are a hot topic for some time now. Here are some tips regarding RAD to get better understanding and maybe solve them. 

For the past few Jumbo versions this issue comes and goes. 

First check if you have any RAD errors and if so, are they impacting you. They could be only cosmetic. Also check how frequent they occur. Start in Smart Console and search for rad alert.

You can open them and get some basic understanding what is going on. In this case below a timeout was reach. You can also see other errors like:

RAD reached maximum allowed concurrent requests

Error occurred while accessing:<URL>

RAD request exceeded maximum handing time,

Second if you experience any issues with browsing and have RAD errors check the following setting in Smart Console:

If you set it to background browsing will continue. If set to Hold / Block browsing is stopped. You can consider changing it to background until RAD issues are solved or more stable. Above is for application control & URL filtering. Same goes for the Threat Prevention blades. 

Third is to check the current version. For debugging you need at least take 101 on R81.20 for the following feature:

UPDATE: RAD extended flow information is now logged into a cyclic CSV file - $FWDIR/log/rad_events/rad_flows.csv. This enhancement provides visibility into RAD connections, helping to monitoring and troubleshooting.

Later more about the CSV.

Still I would recommend the latest GA take 105. There are RAD issues solved and not always listed in release note. 

Fourth is to review the rad.conf and make the changes to it. NOTE: if you are not certain or not sure what to do open a TAC case and ask for advise!

Backup the original rad_conf.C file with the command: cp $FWDIR/conf/rad_conf.C $FWDIR/conf/rad_conf.BKP Edit the rad_conf.C file with the command: vi $FWDIR/conf/rad_conf.C Change the values in the file like this

vi /opt/CPsuite-R81.20/fw1/conf

        :urlfs_service_check_seconds (7200)

        :amws_service_check_seconds (7200)

        :cpu_cores_as_number_of_threads (false)

        :number_of_threads (0)

        :threads_to_cores_ratio (0.334)

        :minimal_resources_usage_ratio (0.2)

        :number_of_threads_fast_response (0)

        :number_of_threads_slow_response (0)

        :number_of_threads_zph_response (0)

        :number_of_threads_update (0)

        :queue_max_capacity (4000)

        :debug_traffic (false)

        :use_dns_cache (true)

        :dns_cache_timeout_sec (2)

        :use_ssl_cache (true)

        :cert_file_name ("ca-bundle.crt")

        :cert_type ("CRT")

        :ssl_version ("TLSv1_0")

        :ciphers ("TLSv1")

        :autodebug (false)

        :timeout_events (false)

        :normal_flow_events (false)

        :log_timeouts (false)

        :log_errors (true)

        :number_of_reports (2048)

        :max_repository_multiplier (20)

        :flow_timeout (6)

        :excessive_flow_timeout (120)

        :transfer_timeout_sec (15)

        :max_flows (2000)

        :max_pc_in_reply (0)

        :max_content_length_in_reply (1600000)

        :retry_mechanism_on (true)

        :max_retries (25)

        :retry_peroid_mins (15)

        :happy_eyeballs_timeout (200)

        :large_scale_min_cpus (100)

        :large_scale_max_threads (70)

        :max_threads (32)

        :max_mal_pm_cache_size (100)

)

:queue_max_capacity value is always max_flows *2  so 2000*2=4000

Smaller setups you can consider first max_flows 1000 and queue_max_capacity 200

number_of_threads_slow_response (0) = 0 the amount of CPU cores you see in cpview

After making the change, please run the following command:

# rad_admin stop ; sleep 6 ; rad_admin start

Note in case of Maestro:

After changing the file in one of the SGMs, run the commands below:

# asg_cp2blades /opt/CPsuite-R81.20/fw1/conf/rad_conf.C

# g_all 'rad_admin stop ; sleep 6 ; rad_admin start'

This will copy the new files to the rest of the SGMs and restart the process on each member.

Fifth after changes to rad.conf  and updating the firewall if needed, it is time to check if the RAD errors are still there.

If there are still RAD errors I would recommend to open a TAC case with the following info.

- cpinfo

- rad.conf file output

- copy and zip the error dir in $FWDIR/log/rad_events/

- copy and analyze the CSV file automaticity created (after take 101) 

Without the CSV file it is complicated for TAC to investigate the issue, make sure it is there. 

Good luck!

[the_rock]

Very good tip @Lesley 

[tjoll]

Nice one! I'll definitely save this post.