Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Thiago_Mourao
Employee
Employee

How to allow specific YouTube channels

How to allow specific YouTube channels

 

Security Engineering Brazil

July, 2021

Versão 02

 

 

Author:

Thiago Mourao, SE

 

Revision:

 

Contents

How to allow specific YouTube channels

Goal

Warning

Limitations and Requirements

Topology

Scripts

Step by step

 

 

Goal

The purpose of the document is to describe the steps necessary to make a granular control of the YouTube channel in Check Point's NGFW, thus being possible to allow or block access to specific YouTube channels from their unique identifier.

 

Warning

The script used in this Proof of Concept was created by me (Thiago Mourão) using official reference documents from the API Management API Reference v1.8 (link), for more details and updates to the API available through Check Point, please visit the Check Point API Reference (link).

 

Limitations and Requirements

  • For this lab, version 1.8 of the Check Point management API was used, available from version R81.10 of the Gaia Operating System. For earlier versions, please refer to the corresponding version reference guide to ensure support for the commands used.
  • To list videos from a YouTube channel, it was used in YouTube Data API v3 (link) and for that you will need to create your own key for this API.
  • To do granular control of YouTube video URLs it will be necessary configure the HTTPS inspection (SSL Inspection) on the gateway

 

Topology

In this lab, we used an R81.10 version of Check Point Gaia defined as Security Gateway and Manager (StandAlone) at the same time and they were virtualized on VMware Workstation 16.

 

Virtual Environment:

  • 1 x Check Point StandAlone (Gateway/ Manager) R81.10
  • 1 x Microsoft Windows 10

 

image.png



Scripts

ytcl_watch.sh

This script performs the following steps:

  • Checks whether new YouTube channels have been considered for the “YouTubeChannels-AllowList” object in the format defined as YTCL-Channel_ID
  • If a new channel has been added to the scripts “ytcl_diff.sh” and “ytcl_update.sh” it should be possible, in case there are no errors and the damaged channels have public videos.

#!/bin/bash -f

source /var/opt/CPshrd-R81.10/tmp/.CPprofile.sh

#Script Directory
EXECDIR="/home/admin/ytcl"
APIKEY=$(cat apikey)
MGMTIP="192.168.157.100"

cd $EXECDIR

echo "#### Executing ytcl_watch.sh at $(date -u) ####"
echo "MGMT - Login"
mgmt_cli login --management $MGMTIP api-key $APIKEY > sid_a.txt

echo "Erasing YouTubeChannels-AllowList.list file"
> YouTubeChannels-AllowList.list

echo "Getting information from YouTubeChannels-AllowList Object and saving on file YouTubeChannels-AllowList.list"
mgmt_cli --management $MGMTIP -s sid_a.txt show application-site-group name "YouTubeChannels-AllowList" --format json | jq '.members[] | select( .name | contains("YTCL"))' | jq '.name' | sort > YouTubeChannels-AllowList.list

echo "MGMT - Logout"
mgmt_cli --management $MGMTIP -s sid_a.txt logout --format json | jq -c '.'

echo "Erasing YouTubeChannels-AllowList.diff file"
> YouTubeChannels-AllowList.diff

echo "Creating new YouTubeChannels-AllowList.diff file"
diff <(sort YouTubeChannels-AllowList.list) <(sort YouTubeChannels-AllowList.list1) | grep "<" | sed 's/< //' > YouTubeChannels-AllowList.diff

echo "Checking YouTubeChannels-AllowList.diff file"
if [[ $(cat YouTubeChannels-AllowList.diff | wc -l) -gt 0 ]]

then
echo "Running script $EXECDIR/ytcl_diff.sh to creating .url files"
sh $EXECDIR/ytcl_diff.sh

else

echo "No changes on YouTubeChannels-AllowList.list file"

fi

echo "Copying file $EXECDIR/YouTubeChannels-AllowList.list to $EXECDIR/YouTubeChannels-AllowList.list1"
cp -f $EXECDIR/YouTubeChannels-AllowList.list $EXECDIR/YouTubeChannels-AllowList.list1


if [[ $(sh -c 'ls *.url | grep YTCL' | wc -l) -gt 0 ]]
then
echo "Running script $EXECDIR/ytcl_update.sh to Updating URLs on Group YouTubeChannels-AllowList"
sh $EXECDIR/ytcl_update.sh
else
echo "No YTCL.url files were found"
fi

echo "Backup last URL files"
sh -c 'ls *.url | grep YTCL' | sed 's/\(YTCL.*\)\.url/\1/' | while read NAME
do
mv -f ${NAME}.url ${NAME}.bkp-$(date +%Y%m%d%H%M%S)
done

echo "#### End of Execution of ytcl_watch.sh at $(date -u) ####"


 

ytcl_diff.sh

This script performs the following steps:

  • Check if the channel ID exists and if it has public videos
  • If the Channel exists and has public videos, recursively, using a YouTube Data API v3, all URLs of public videos are extracted and saved in a file.

#!/bin/bash -f

source /var/opt/CPshrd-R81.10/tmp/.CPprofile.sh

#Script Directory
EXECDIR="/home/admin/ytcl"

#Google API Variables
GAPIKEY=$(cat gapikey)
CID=""
MR=5
NTP_S=""

echo "#### Executing ytcl_diff.sh at $(date -u) ####"

cat YouTubeChannels-AllowList.diff | sed 's/"YTCL\-\(.*\)"/\1/' | while read CID
do
if [[ $(curl_cli -s -k "https://www.googleapis.com/youtube/v3/search?key=$GAPIKEY&channelId=$CID&part=snippet,id&order=date&..." | jq '.pageInfo.totalResults') -gt 0 ]]
then
echo "Erasing file YTCL-$CID.url"
> YTCL-${CID}.url
curl_cli -k -s "https://www.googleapis.com/youtube/v3/search?key=$GAPIKEY&channelId=$CID&part=snippet,id&order=date&..." > result-${CID}.txt
while true
do
if [[ $(cat result-${CID}.txt | jq '.' | grep "nextPageToken" | wc -l) -gt 0 ]]
then
echo "nextPageToken ($NTP_S) found for channel $CID"
NTP_S="$(cat result-${CID}.txt | jq '.' | grep nextPageToken | cut -d '"' -f 4)"
echo "Adding URLs to file YTCL-${CID}.urls - nextPageToken will be $NTP_S"
cat result-${CID}.txt | jq '.items[].id.videoId' | grep -v null | sed 's/^\"/\"www\.youtube\.com\/watch\?v\=/g' >> YTCL-${CID}.url
curl_cli -k -s "https://www.googleapis.com/youtube/v3/search?key=$GAPIKEY&channelId=$CID&part=snippet,id&order=date&..." > result-${CID}.txt
else
echo "nextPageToken not found or last page for channel $CID"
echo "Adding last page of URLs to file YTCL-$CID.urls"
cat result-${CID}.txt | jq '.items[].id.videoId' | grep -v null | sed 's/^\"/\"www\.youtube\.com\/watch\?v\=/g' >> YTCL-${CID}.url
echo "File $CID.url finished"
break
fi
done
else
echo "YouTube Channel $CID - No public videos found or channel ID is wrong"
fi
done

echo "#### End of Execution of ytcl_diff.sh at $(date -u) ####"


ytcl_update.sh

 

This script performs the following steps:

  • Updates the corresponding YouTube Channel object with the channel title in the “COMMENTS” field, which can be used as a search parameter in SmartConsole in the object tree
  • Check which URLs of public videos identified in the previous process have not yet been registered in the corresponding channel object and add them.

#!/bin/bash -f

source /var/opt/CPshrd-R81.10/tmp/.CPprofile.sh

#Script Directory
EXECDIR="/home/admin/ytcl"

#Check Point Management Variables
APIKEY=$(cat apikey)
MGMTIP="192.168.157.100"

#Google API Variables
GAPIKEY=$(cat gapikey)

cd $EXECDIR

echo "#### Executing ytcl_update.sh at $(date -u) ####"

echo "MGMT - Login"
mgmt_cli login --management $MGMTIP api-key $APIKEY > sid_b.txt


sh -c 'ls *.url' |grep YTCL | sed 's/YTCL\-\(.*\)\.url/\1/' | while read NAME
do
mgmt_cli --management $MGMTIP -s sid_b.txt show application-site name "YTCL-${NAME}" --format json | jq '."url-list"[]' | sort > YTCL-${NAME}.list

echo "Creating new YTCL-${NAME}.diff file"
diff YTCL-${NAME}.url YTCL-${NAME}.list | grep "<" | sed 's/< //' > YTCL-${NAME}.diff

echo "Checking YTCL-${NAME}.diff file"
if [[ $(cat YTCL-${NAME}.diff | wc -l) -gt 0 ]]
then
echo "Updating Channel Title on Object comments"
CTITLE=$(curl_cli -k -s "https://www.googleapis.com/youtube/v3/search?key=$GAPIKEY&channelId=${NAME}&part=snippet,id&order=da..." | jq '.items[].snippet.channelTitle')
echo "mgmt_cli --management $MGMTIP -s sid_b.txt set application-site name \"YTCL-${NAME}\" comments $CTITLE --format json"
mgmt_cli --management $MGMTIP -s sid_b.txt set application-site name "YTCL-${NAME}" comments "$CTITLE" --format json | jq -c '.'

echo "Reading file YTCL-${NAME}.diff and update object ${NAME}"
cat YTCL-${NAME}.diff | while read URL
do
mgmt_cli --management $MGMTIP -s sid_b.txt set application-site name "YTCL-${NAME}" url-list.add $URL --format json | jq -c '.'
done
else
echo "No changes on YTCL-${NAME}.diff file"
fi

done

echo "MGMT - Publishing Session"
mgmt_cli --management $MGMTIP -s sid_b.txt publish --format json | jq -c '.'

echo "MGMT - Pushing Policy"
mgmt_cli --management $MGMTIP -s sid_b.txt install-policy policy-package "standard" access true threat-prevention false targets.1 "R81.10-StandAlone" --format json | jq -c '.'

echo "MGMT - Logout"
mgmt_cli --management $MGMTIP -s sid_b.txt logout --format json | jq -c '.'

echo "#### End of Execution of ytcl_update.sh at $(date -u) ####"


 

Step by step

  1. Enable the Functions: “Application Control” and “URL Filtering”.
    image.png
  2. Enable the SSL Inspection Functionality “Enable HTTPS Inspection”.
    image.png
  3. Create the “YouTube-Domain” object
    • Add YouTube domain: \.youtube\.com
    • Check the option “URLs are defined as Regular Expression
      image.png
  4. Create the object “Application/Site Group” and name it “YouTubeChannels-AllowList”
    • Add the YouTube domain: \.youtube\.com
    • Check the option “URLs are defined as Regular Expression
      image.png
  5. Add to the group “YouTubeChannels-AllowList”
    • Click on the + button
    • Click on the * button
    • Select the option “Application/Site...
      image.png
  6. Configure the group using the following pattern
  7. Configure access rules for:
    • Blocking the QUIC protocol
    • Web Filtering access rule with Inline Layer (Application Control & URL Filtering)
      • Allow rule for the YouTube Channel List (YouTubeChannels-AllowList)
      • Block rule for the YouTube Domain (YouTube-Domain)
      • Allow rule for the YouTube Application
        image.png

        image.png

        image.png
  8. Copy the 03 (three) script files to the Check Point Management Server
    image.png
  9. Create the apikey file containing the Check Point Management user API key and the gpaikey file containing the Google API key to query via YouTube Data API v3
    image.png
  10. Run the ytcl_watch.sh script
    image.png 
    1. Check the content of the object “YouTubeChannels-AllowList” and look for Application/Site that start with the prefix “YTCL-“ and save it on the file “YouTubeChannels-AllowList.list”.
      image.png

      image.png
    2. Compares the contents of the file “YouTubeChannels-AllowList.list” with the file “YouTubeChannels-AllowList.list1” to see if any new channels have been added and generates the file “YouTubeChannels-AllowList.diff
      image.png
    3. If the “YouTubeChannels-AllowList.diff” file has any information it will run a new script named “ytcl_diff.sh” to create the YTCL-{Channel_ID}.url files with the URLs of all the public videos of each new chaneel that was added to the “YouTubeChannels-AllowList” object
      image.png
    4. Automatic execution of the ytcl_diff.sh script to create the {Channel_ID}.url files, if the “YouTubeChannels-AllowList.diff” file is not empty
    5. The contents of the “YouTubeChannels-AllowList” file will be moved to the “YouTubeChannels-AllowList1” file to be used as a comparison basis for the next script execution.
    6. Automatic execution of the ytcl_update.sh script if there are YTCL-{Channel_ID}.url files in the directory to update the objects referring to YouTube Channels with the list of URLs that are not yet registered, in addition to updating the COMMENTS field with the channel title.
    7. The script moves the YTCL-{Channel_ID}.url files generated in the previous step to the YTCL-{Channel_ID}.bkp-$(date +%Y%m%d%H%M%S) format to serve as a history of which URLs were added at that time
  11. Automatic execution of the ytcl_diff.shscript
    image.png 
    1. For each line of the “YouTubeChannels-AllowList.diff” the following actions are performed:
    2. Parsing to find the YouTube Channel ID, as defined in the creation of objects of the “Application/Site” type following the YTCL-{Channel_ID} rule
    3. Delete, if any, the contents of the YTCL-{Channel_ID}.url file
    4. Check if the Channel ID is valid and if the Channel has public video
    5. Recursively list all the channel's public video URLs and save to {Channel_ID}.url file
      image.png

      image.png
  12. Automatic execution of the ytcl_diff.sh script
    image.png
    image.png
    image.png
    1. Extracts the YouTube channel ID that was added to the “YouTubeChannels-AllowList” object as per specified naming YTCL-{Channel_ID}
    2. Query the object of type “Application/Site” added to the group “YouTubeChannels-AllowList” and save the current URL list of each object in a file “YTCL-{Channel_ID}.list
      image.png
    3. Checks the difference between the files “YTCL-{Channel_ID}.url”, generated by the script ytcl_diff.sh and the newly created file “YTCL-{Channel_ID}.list” and saves the additional URLs of the file “YTCL-{Channel_ID}.url” in another “YTCL-{Channel_ID}.diff file
      image.png
    4. Updates the “Comments” field of the “YTCL-{Channel_ID}” object with the corresponding YouTube Channel Title
      image.png

      image.png
    5. Updates the list of URLs of the object “YTCL-{Channel_ID}” with the additional URLs saved in the file “YTCL-{Channel_ID}.diff
      image.png
    6. Publish the session
    7. Install the Policy
  13. List of files in directory after execution
    image.png 
    •  

 

 

 

(1)
0 Replies