Reserve a CPU for custom bash scripts

[redcrow]

Hello,

we have some bash scripts that fetch external security feeds and populate dynamic objects, which are then used in rulebase to block a lot of malicious IPs.

Since these scripts execute several heavy actions, we'd like to assign a specific CPU only for such bash scripts so that CP processes running on other CPUs are not interested.

We have 3 x 23900 appliances with 36 cores. The architecture is configured in VSX with 4 VS: Active/Standby/Backup. The first 4 cores are assigned to interfaces, the other 32 are for all VS processes.

At present the configuration is the following:
- VS1 has 10 instances assigned
- VS2 has 8 instances assigned
- VS4 has 10 instances assigned
- VS5 has 1 instances assigned

[Expert@lntfw-pgtw1:0]# fw ctl affinity -l
Mgmt: CPU 0
Sync: CPU 1
eth1-01: CPU 2
eth1-02: CPU 3
eth1-03: CPU 3
eth1-04: CPU 2
eth2-01: CPU 1
eth2-02: CPU 0
eth2-03: CPU 0
eth2-04: CPU 1
eth2-05: CPU 2
eth2-06: CPU 3
eth2-07: CPU 3
eth2-08: CPU 2
VS_0 fwk: CPU 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
VS_1 fwk: CPU 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
VS_2 fwk: CPU 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
VS_3 fwk: CPU 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
VS_4 fwk: CPU 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
VS_5 fwk: CPU 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
Interface eth3-01: has multi queue enabled
Interface eth3-02: has multi queue enabled

I know CPU reservation to specific process is possible in Linux (e.g. isolcpus). How can we do that in GAIA OS?

Thanks.

[PhoneBoy]

Pretty sure this isn't possible at the moment.

[Timothy_Hall]

What I would suggest is to increase the CPU priority of your script to near the maximum allowable, so that the script will have priority over almost all other processes in the system.

So if your script name is "scriptname.sh" you invoke it with elevated priority like this:

nice --19 ./scriptname.sh

You can confirm the elevated priority by looking at the "NI" column in top while the process is running.  -20 is the maximum value but is used by various kernel threads which you don't want to interfere with if somehow your script gets stuck and runs away with the CPU.  In addition, I'd strongly recommend adding logic at the start of the script making sure there is not already another instance running at the time of invocation.  This will avoid a situation where there are many instances of the script inappropriately executing with high CPU priority, which can slam all the CPUs thus starving out any lower priority processes, and bring the system to what looks like a near halt.

[redcrow]

@Timothy_Hall 

Thank you very much for this alternative solution. Anyway, if I increase the CPU priority for my bash scripts and that CPU at that moment is used by, for instance, a fwk process... could that be a problem? I think we could occur in performance issues, or am I wrong?

[Kaspars_Zibarts]

Never played myself but you should be able to use i.e (where X is the desired core)

taskset --cpu-list X my_script.sh

You just need to adjust your VSX pool not to use that one particular core for SND nor FWK

Might work 🙂

[Timothy_Hall]

The Gaia OS should try to seek out an idle CPU first, and once the script starts it won't be allowed to completely monopolize a single CPU, but will be given much higher scheduling priority (i.e. more CPU slices).  Even with elevated priority the script can be shuffled around to different CPUs that are idle, so impact against fwk processes should be minimal. 

So once again the only caveat is to ensure at the start of the script that there already isn't another copy of it running and you should be OK.  Typical daemons do this by writing their Process ID (PID) into a known file in someplace like /tmp when they start up, and when the script starts it checks for the presence of the PID file, and if it exists, see if the stated PID value is associated with an existing process.  If so the script terminates immediately. 

When the script successfully finishes execution, the PID file is removed.  Even if it somehow crashes/aborts without removing the PID file, this technique works to keep multiple instances from running.