IMPORTANT SECURITY NOTICE FOR CUSTOMERS USING KUBEFLOW

[Maya_Levine]

Kubeflow is an open source machine learning toolkit for running machine learning tasks in Kubernetes. Nodes that are used for ML tasks are often relatively powerful, sometimes including GPUs. This makes Kuberenetes clusters used for ML tasks a perfect target for crypto mining campaigns. Azure Security Center has uncovered a new attack that specifically targets Kubeflow environments. In this attack, Kubeflow was most likely used as the access vector to deploy an image from a public repository to many different clusters. The image is ddsfdfsaadfs/dfsdf:99 and contains an XMRIG miner.

       

                                                       Inspecting this image’s layers reveals that this image runs an XMRIG miner

Customers with Kubeflow deployed in clusters should take the following steps immediately to protect themselves:

1. Verify that the malicious container is not deployed in the cluster. The following command can help you to check it:

kubectl get pods –all-namespaces -o jsonpath=”{.items[*].spec.containers[*].image}”  | grep -i ddsfdfsaadfs 

2. Make sure the Kubeflow dashboard is not exposed to the internet: check the type of the Isto ingress service with the following command and make sure that it is not a load balancer with a public IP:

kubectl get service istio-ingressgateway -n istio-system

3. Set up a Continuous Compliance rule in CloudGuard Dome9 to be notified if any of your containers contain this malicious image.

KubernetesPod should not have spec.containers contain-any [ image like '%ddsfdfsaadfs%' ]

DETAILED STEPS:

1. Login to the Dome9 console and click on Posture Management. You should be in the Compliance Ruleset tab. Click on ADD RULESET. Fill in the Name and Description, select Kubernetes as the platform, and click CREATE.

2. Search for the RuleSet once it is created and click on NEW RULE

3. Click on the box under Rule GSL. Change it from Builder to Free Text and paste the following GSL rule into the box:

KubernetesPod should not have spec.containers contain-any [ image like '%ddsfdfsaadfs%' ]

Select the relevant account and click on DONE

Name the Rule and click Save. Repeat Step 3 for any other accounts you want to protect.

4. Click on ADD POLICY and Cloud Account Policy to set up Notifications for whenever this rule fails on your accounts.

Select the relevant accounts and press NEXT

Select a previously defined notification, or to create a new one click ADD NOTIFICATION. Select the notification options for the policy, as follows:

• Alerts Console - each finding for this policy will be sent to the Alerts page (in Notifications, in the Administration menu)
• Scheduled Report - a report will be sent to email recipients regular periods. Select the time and frequency of the report, and the type (summary or detailed). Enter a list of email recipients for the report.
• Immediate Notification - a notification will be sent for each new or changed finding. Select the type of notification.
  • For email notifications, enter a list of email recipients.
  • For SNS notifications, enter the ARN for the AWS SNS topic, and select the format for the notification:
    • JSON - Full entity includes details of the finding, and full attributes (as maintained in Dome9) for the entity in the finding, in JSON format
    • JSON  - Basic entity includes details of the finding, and a few attributes for the entity (such as the entity id), in JSON format
    • Plain text - like the Basic entity, but in plain text format.
      Click Send test message to test the connection
  • For HTTP Endpoint notifications, enter the URL for the endpoint, and select the authentication that is used. For Basic authentication, enter also the username & password. Select also the format of the notification:
    • JSON - Full entity - this is the default selection
    • Splunk - JSON - select this for Splunk endpoints

Notifications to HTTP endpoints will be issued from one of these fixed IP addresses: 3.232.156.115, 52.70.61.156, or 3.231.193.67