topic Re: Skyline + VSX: "OpenTelemetry Components are not up yet" in OpenTelemetry/Skyline

Skyline + VSX: "OpenTelemetry Components are not up yet"

Toolmaker — Fri, 10 Feb 2023 15:48:47 GMT

Hi,

we try to use Skyline on R81.10, following sk178566. Our version is

Product version Check Point Gaia R81.10
OS build 335
OS kernel version 3.10.0-957.21.3cpx86_64
OS edition 64-bit

Running '/opt/CPotelcol/REST.py --set_open_telemetry "$(cat payload.json)"' throws an

Exception: OpenTelemetry Components are not up yet

This seems to be due to the result from '/opt/CPviewExporter/CPviewExporterCli.sh show':

{
"active_after_reboot":"false",
"status":"Agents are down"
}

Looking into CPviewExporterCli.sh, we see that the "show" parameter branches into the function product_status_json.

I assume that this function should check each virtual systems (loop over $(vslist) ) and either return "Agents are down", "All agents are up" or "Agents are partially up". To that end, it sets state variables FOUND_ONE_NON_ACTIVE and FOUND_ACTIVE:

function product_status_json() {
 for VS in $(vslist); do
     STAT=$(${COMPONENT_DIR}/otlp_wd.bash -o stat ${VS})
     FOUND_ONE_NON_ACTIVE=false
     FOUND_ACTIVE=false
     if [[ ${STAT} == "Agent is not running" ]]; then
       FOUND_ONE_NON_ACTIVE=true
     elif [[ ${STAT} == "Agent is running" ]]; then
       FOUND_ACTIVE=true
     fi
     if ${FOUND_ONE_NON_ACTIVE} && ${FOUND_ACTIVE}; then
       RC_EXPORTER="Agents are partially up"
     elif ${FOUND_ONE_NON_ACTIVE}; then
       RC_EXPORTER="Agents are down"
     elif ${FOUND_ACTIVE}; then
       RC_EXPORTER="All agents are up"
      else
        RC_EXPORTER="Unknown"
     fi
  done

  echo "{" 
  if [[ $(is_product_active) -eq 1 ]]; then
    echo -e "\t\"active_after_reboot\":\"true\","
  else
    echo -e "\t\"active_after_reboot\":\"false\","
  fi
  echo -e "\t\"status\":\"${RC_EXPORTER}\""
  echo "}"
}

However, the state variables get set to default values inside the for loop (instead of before), and the evaluation logic setting RC_EXPORTER also runs inside the loop (instead of after). I wonder if this is really intended?

As a consequence, the final result only depends on the VS with the highest number, not on any of the other VSs; and "Agents are partially up" will never be a result.

At least, that is my interpretation... I might also have misunderstood this functions intent or working.

In our setup, the highest-numbered VS is a virtual switch for which the test in line 3 of product_status_json()

otlp_wd.bash -o stat ${VS})

returns "not running", so the above /opt/CPotelcol/REST.py will always fail.

After that lengthy prologue, I wonder:

- Is the above behaviour correct in CPviewExporterCli.sh?
- Can/should we activate the CpviewExporter agent in the virtual switches?
- If so, how?
- or is it safe to modify "CPviewExporterCli.sh show" to always return "All agents are up"?

Many thanks,

Bernhard

Re: Skyline + VSX: "OpenTelemetry Components are not up yet"

Chris_Atkinson — Fri, 10 Feb 2023 23:07:57 GMT

To confirm the system in question is running Jumbo take 79 or higher? To check run:

cpinfo -y all

Re: Skyline + VSX: "OpenTelemetry Components are not up yet"

JozkoMrkvicka — Sat, 11 Feb 2023 18:03:28 GMT

Not sure who is the original code owner, but looks like the logic explained is pretty clear and the code should be revised.

Looping in @Arik_Ovtracht

Re: Skyline + VSX: "OpenTelemetry Components are not up yet"

the_rock — Sat, 11 Feb 2023 20:05:57 GMT

I followed the sk you mentioned using Grafana method and worked fine. I tested this on R81.10 jumbo 87, which is the latest, but as @Chris_Atkinson mentioned, have you ensured you are on at least take 79 as indicated in the article?

Just run cpinfo -y FW1 and it will give you the jumbo version.

Andy

Re: Skyline + VSX: "OpenTelemetry Components are not up yet"

Elad_Chomsky — Sun, 12 Feb 2023 11:14:37 GMT

@Toolmaker wrote:

Hi,

we try to use Skyline on R81.10, following sk178566. Our version is
Product version Check Point Gaia R81.10
OS build 335
OS kernel version 3.10.0-957.21.3cpx86_64
OS edition 64-bit
Running '/opt/CPotelcol/REST.py --set_open_telemetry "$(cat payload.json)"' throws an

Exception: OpenTelemetry Components are not up yet

This seems to be due to the result from '/opt/CPviewExporter/CPviewExporterCli.sh show':
{
"active_after_reboot":"false",
"status":"Agents are down"
}
Looking into CPviewExporterCli.sh, we see that the "show" parameter branches into the function product_status_json.

I assume that this function should check each virtual systems (loop over $(vslist) ) and either return "Agents are down", "All agents are up" or "Agents are partially up". To that end, it sets state variables FOUND_ONE_NON_ACTIVE and FOUND_ACTIVE:
function product_status_json() {
 for VS in $(vslist); do
     STAT=$(${COMPONENT_DIR}/otlp_wd.bash -o stat ${VS})
     FOUND_ONE_NON_ACTIVE=false
     FOUND_ACTIVE=false
     if [[ ${STAT} == "Agent is not running" ]]; then
       FOUND_ONE_NON_ACTIVE=true
     elif [[ ${STAT} == "Agent is running" ]]; then
       FOUND_ACTIVE=true
     fi
     if ${FOUND_ONE_NON_ACTIVE} && ${FOUND_ACTIVE}; then
       RC_EXPORTER="Agents are partially up"
     elif ${FOUND_ONE_NON_ACTIVE}; then
       RC_EXPORTER="Agents are down"
     elif ${FOUND_ACTIVE}; then
       RC_EXPORTER="All agents are up"
      else
        RC_EXPORTER="Unknown"
     fi
  done

  echo "{" 
  if [[ $(is_product_active) -eq 1 ]]; then
    echo -e "\t\"active_after_reboot\":\"true\","
  else
    echo -e "\t\"active_after_reboot\":\"false\","
  fi
  echo -e "\t\"status\":\"${RC_EXPORTER}\""
  echo "}"
}
However, the state variables get set to default values inside the for loop (instead of before), and the evaluation logic setting RC_EXPORTER also runs inside the loop (instead of after). I wonder if this is really intended?

As a consequence, the final result only depends on the VS with the highest number, not on any of the other VSs; and "Agents are partially up" will never be a result.

At least, that is my interpretation... I might also have misunderstood this functions intent or working.

In our setup, the highest-numbered VS is a virtual switch for which the test in line 3 of product_status_json()

otlp_wd.bash -o stat ${VS})

returns "not running", so the above /opt/CPotelcol/REST.py will always fail.

After that lengthy prologue, I wonder:

- Is the above behaviour correct in CPviewExporterCli.sh?
- Can/should we activate the CpviewExporter agent in the virtual switches?
- If so, how?
- or is it safe to modify "CPviewExporterCli.sh show" to always return "All agents are up"?

Many thanks,

Bernhard

Hi @Toolmaker,

We have found the issue, it seems like this is happening when the last VS has a different status then the ones before it. We will fix the code, and it will be pushed as part of the next version. Out of curiosity, can you expend on why the last VS has a different status then the previous ones?

You can modify the script as follows to fix this issue:

for VS in $(vslist); do STAT=$(${COMPONENT_DIR}/otlp_wd.bash -o stat ${VS}) FOUND_ONE_NON_ACTIVE=false FOUND_ACTIVE=false if [[ ${STAT} == "Agent is not running" ]]; then FOUND_ONE_NON_ACTIVE=true elif [[ ${STAT} == "Agent is running" ]]; then FOUND_ACTIVE=true fi if ${FOUND_ONE_NON_ACTIVE} && ${FOUND_ACTIVE}; then RC_EXPORTER="Agents are partially up" elif ${FOUND_ONE_NON_ACTIVE}; then RC_EXPORTER="Agents are down" elif ${FOUND_ACTIVE}; then RC_EXPORTER="All agents are up" else RC_EXPORTER="Unknown" fi done

Change to:

FOUND_ONE_NON_ACTIVE=false FOUND_ONE_ACTIVE=false for VS in $(vslist); do STAT=$(${COMPONENT_DIR}/otlp_wd.bash -o stat ${VS}) if [[ ${STAT} == "Agent is not running" ]]; then FOUND_ONE_NON_ACTIVE=true elif [[ ${STAT} == "Agent is running" ]]; then FOUND_ONE_ACTIVE=true fi done if ${FOUND_ONE_NON_ACTIVE} && ${FOUND_ONE_ACTIVE}; then RC_EXPORTER="Agents are partially up" elif ${FOUND_ONE_NON_ACTIVE}; then RC_EXPORTER="Agents are down" elif ${FOUND_ONE_ACTIVE}; then RC_EXPORTER="All agents are up" else RC_EXPORTER="Unknown" fi

Re: Skyline + VSX: "OpenTelemetry Components are not up yet"

Toolmaker — Mon, 13 Feb 2023 09:36:08 GMT

Take 81:

[Expert@fw-vsxa-01:0]# cpinfo -y FW1
This is Check Point CPinfo Build 914000231 for GAIA
[FW1]
HOTFIX_R81_10_JUMBO_HF_MAIN Take: 81

Re: Skyline + VSX: "OpenTelemetry Components are not up yet"

NUNO_C — Tue, 30 May 2023 07:23:23 GMT

Hi,

After changing the script, the result went from
"Agent is not running"

to
"Agents are partially up"

Which is expected since the vsx environment im testing has VS switches

But /opt/CPotelcol/REST.py

Expects

"All agents are up"

So "Agents are partially up" doesnt do the job as expected

if len(filter) == 0 or 'exporter' in filter:
status.append(json.loads(out_cpview_exporter)["status"] == "All agents are up")

Cheers,

Nuno

Re: Skyline + VSX: "OpenTelemetry Components are not up yet"

Elad_Chomsky — Tue, 30 May 2023 09:21:27 GMT

Hi @NUNO_C ,

Please download REST.py from here, (SHA1 should start with '55da7f....', please contact me on eladch@checkpoint.com, if SHA1 on the website is mismatched ), Replace /opt/CPotelcol/REST.py with it , and retry to run the script. This is a newer version, that is going to be pushed as part of an upcoming AutoUpdater release ( Component name is CPotelcol ).

Re: Skyline + VSX: "OpenTelemetry Components are not up yet"

NUNO_C — Fri, 02 Jun 2023 07:44:34 GMT

Hi @Elad_Chomsky ,

The new REST.py and CPviewExporterCli.sh modification did the job, skyline telemetry working as expected.

If we can get a notification in which take will be added would be great.

Thanks,
N

Re: Skyline + VSX: "OpenTelemetry Components are not up yet"

rizkyw — Fri, 19 Apr 2024 11:16:31 GMT

Hi, I cant downlod REST.py file. with below information

Missing software subscription to download this file.

Thanks

Re: Skyline + VSX: "OpenTelemetry Components are not up yet"

Chris_Atkinson — Fri, 19 Apr 2024 14:24:08 GMT

Do you have an account associated with a valid support contract?

Re: Skyline + VSX: "OpenTelemetry Components are not up yet"

Elad_Chomsky — Tue, 23 Apr 2024 14:36:32 GMT

Hi @rizo

Please see the official sk, we moved to a new tool ( sklnctl ), please try to see if using it resolves the issue.

Re: Skyline + VSX: "OpenTelemetry Components are not up yet"

rizkyw — Tue, 07 May 2024 06:31:52 GMT

Hi Chris,

I guess my account is not associated with valid support. Do you have any solutions? because in grafana for vsx not showing the data. I already attached screenshot for vsx0, vsx1, and vsx2