How to allow specific YouTube channels
Security Engineering Brazil
July, 2021
Versão 02
Author:
Thiago Mourao, SE
Revision:
Contents
How to allow specific YouTube channels
Goal
Warning
Limitations and Requirements
Topology
Scripts
Step by step
Goal
The purpose of the document is to describe the steps necessary to make a granular control of the YouTube channel in Check Point's NGFW, thus being possible to allow or block access to specific YouTube channels from their unique identifier.
Warning
The script used in this Proof of Concept was created by me (Thiago Mourão) using official reference documents from the API Management API Reference v1.8 (link), for more details and updates to the API available through Check Point, please visit the Check Point API Reference (link).
Limitations and Requirements
- For this lab, version 1.8 of the Check Point management API was used, available from version R81.10 of the Gaia Operating System. For earlier versions, please refer to the corresponding version reference guide to ensure support for the commands used.
- To list videos from a YouTube channel, it was used in YouTube Data API v3 (link) and for that you will need to create your own key for this API.
- To do granular control of YouTube video URLs it will be necessary configure the HTTPS inspection (SSL Inspection) on the gateway
Topology
In this lab, we used an R81.10 version of Check Point Gaia defined as Security Gateway and Manager (StandAlone) at the same time and they were virtualized on VMware Workstation 16.
Virtual Environment:
- 1 x Check Point StandAlone (Gateway/ Manager) R81.10
- 1 x Microsoft Windows 10
Scripts
ytcl_watch.sh
This script performs the following steps:
- Checks whether new YouTube channels have been considered for the “YouTubeChannels-AllowList” object in the format defined as YTCL-Channel_ID
- If a new channel has been added to the scripts “ytcl_diff.sh” and “ytcl_update.sh” it should be possible, in case there are no errors and the damaged channels have public videos.
|
#!/bin/bash -f
source /var/opt/CPshrd-R81.10/tmp/.CPprofile.sh
#Script Directory
EXECDIR="/home/admin/ytcl"
APIKEY=$(cat apikey)
MGMTIP="192.168.157.100"
cd $EXECDIR
echo "#### Executing ytcl_watch.sh at $(date -u) ####"
echo "MGMT - Login"
mgmt_cli login --management $MGMTIP api-key $APIKEY > sid_a.txt
echo "Erasing YouTubeChannels-AllowList.list file"
> YouTubeChannels-AllowList.list
echo "Getting information from YouTubeChannels-AllowList Object and saving on file YouTubeChannels-AllowList.list"
mgmt_cli --management $MGMTIP -s sid_a.txt show application-site-group name "YouTubeChannels-AllowList" --format json | jq '.members[] | select( .name | contains("YTCL"))' | jq '.name' | sort > YouTubeChannels-AllowList.list
echo "MGMT - Logout"
mgmt_cli --management $MGMTIP -s sid_a.txt logout --format json | jq -c '.'
echo "Erasing YouTubeChannels-AllowList.diff file"
> YouTubeChannels-AllowList.diff
echo "Creating new YouTubeChannels-AllowList.diff file"
diff <(sort YouTubeChannels-AllowList.list) <(sort YouTubeChannels-AllowList.list1) | grep "<" | sed 's/< //' > YouTubeChannels-AllowList.diff
echo "Checking YouTubeChannels-AllowList.diff file"
if [[ $(cat YouTubeChannels-AllowList.diff | wc -l) -gt 0 ]]
then
echo "Running script $EXECDIR/ytcl_diff.sh to creating .url files"
sh $EXECDIR/ytcl_diff.sh
else
echo "No changes on YouTubeChannels-AllowList.list file"
fi
echo "Copying file $EXECDIR/YouTubeChannels-AllowList.list to $EXECDIR/YouTubeChannels-AllowList.list1"
cp -f $EXECDIR/YouTubeChannels-AllowList.list $EXECDIR/YouTubeChannels-AllowList.list1
if [[ $(sh -c 'ls *.url | grep YTCL' | wc -l) -gt 0 ]]
then
echo "Running script $EXECDIR/ytcl_update.sh to Updating URLs on Group YouTubeChannels-AllowList"
sh $EXECDIR/ytcl_update.sh
else
echo "No YTCL.url files were found"
fi
echo "Backup last URL files"
sh -c 'ls *.url | grep YTCL' | sed 's/\(YTCL.*\)\.url/\1/' | while read NAME
do
mv -f ${NAME}.url ${NAME}.bkp-$(date +%Y%m%d%H%M%S)
done
echo "#### End of Execution of ytcl_watch.sh at $(date -u) ####"
|
ytcl_diff.sh
This script performs the following steps:
- Check if the channel ID exists and if it has public videos
- If the Channel exists and has public videos, recursively, using a YouTube Data API v3, all URLs of public videos are extracted and saved in a file.
|
#!/bin/bash -f
source /var/opt/CPshrd-R81.10/tmp/.CPprofile.sh
#Script Directory
EXECDIR="/home/admin/ytcl"
#Google API Variables
GAPIKEY=$(cat gapikey)
CID=""
MR=5
NTP_S=""
echo "#### Executing ytcl_diff.sh at $(date -u) ####"
cat YouTubeChannels-AllowList.diff | sed 's/"YTCL\-\(.*\)"/\1/' | while read CID
do
if [[ $(curl_cli -s -k "https://www.googleapis.com/youtube/v3/search?key=$GAPIKEY&channelId=$CID&part=snippet,id&order=date&..." | jq '.pageInfo.totalResults') -gt 0 ]]
then
echo "Erasing file YTCL-$CID.url"
> YTCL-${CID}.url
curl_cli -k -s "https://www.googleapis.com/youtube/v3/search?key=$GAPIKEY&channelId=$CID&part=snippet,id&order=date&..." > result-${CID}.txt
while true
do
if [[ $(cat result-${CID}.txt | jq '.' | grep "nextPageToken" | wc -l) -gt 0 ]]
then
echo "nextPageToken ($NTP_S) found for channel $CID"
NTP_S="$(cat result-${CID}.txt | jq '.' | grep nextPageToken | cut -d '"' -f 4)"
echo "Adding URLs to file YTCL-${CID}.urls - nextPageToken will be $NTP_S"
cat result-${CID}.txt | jq '.items[].id.videoId' | grep -v null | sed 's/^\"/\"www\.youtube\.com\/watch\?v\=/g' >> YTCL-${CID}.url
curl_cli -k -s "https://www.googleapis.com/youtube/v3/search?key=$GAPIKEY&channelId=$CID&part=snippet,id&order=date&..." > result-${CID}.txt
else
echo "nextPageToken not found or last page for channel $CID"
echo "Adding last page of URLs to file YTCL-$CID.urls"
cat result-${CID}.txt | jq '.items[].id.videoId' | grep -v null | sed 's/^\"/\"www\.youtube\.com\/watch\?v\=/g' >> YTCL-${CID}.url
echo "File $CID.url finished"
break
fi
done
else
echo "YouTube Channel $CID - No public videos found or channel ID is wrong"
fi
done
echo "#### End of Execution of ytcl_diff.sh at $(date -u) ####"
|
ytcl_update.sh
This script performs the following steps:
- Updates the corresponding YouTube Channel object with the channel title in the “COMMENTS” field, which can be used as a search parameter in SmartConsole in the object tree
- Check which URLs of public videos identified in the previous process have not yet been registered in the corresponding channel object and add them.
|
#!/bin/bash -f
source /var/opt/CPshrd-R81.10/tmp/.CPprofile.sh
#Script Directory
EXECDIR="/home/admin/ytcl"
#Check Point Management Variables
APIKEY=$(cat apikey)
MGMTIP="192.168.157.100"
#Google API Variables
GAPIKEY=$(cat gapikey)
cd $EXECDIR
echo "#### Executing ytcl_update.sh at $(date -u) ####"
echo "MGMT - Login"
mgmt_cli login --management $MGMTIP api-key $APIKEY > sid_b.txt
sh -c 'ls *.url' |grep YTCL | sed 's/YTCL\-\(.*\)\.url/\1/' | while read NAME
do
mgmt_cli --management $MGMTIP -s sid_b.txt show application-site name "YTCL-${NAME}" --format json | jq '."url-list"[]' | sort > YTCL-${NAME}.list
echo "Creating new YTCL-${NAME}.diff file"
diff YTCL-${NAME}.url YTCL-${NAME}.list | grep "<" | sed 's/< //' > YTCL-${NAME}.diff
echo "Checking YTCL-${NAME}.diff file"
if [[ $(cat YTCL-${NAME}.diff | wc -l) -gt 0 ]]
then
echo "Updating Channel Title on Object comments"
CTITLE=$(curl_cli -k -s "https://www.googleapis.com/youtube/v3/search?key=$GAPIKEY&channelId=${NAME}&part=snippet,id&order=da..." | jq '.items[].snippet.channelTitle')
echo "mgmt_cli --management $MGMTIP -s sid_b.txt set application-site name \"YTCL-${NAME}\" comments $CTITLE --format json"
mgmt_cli --management $MGMTIP -s sid_b.txt set application-site name "YTCL-${NAME}" comments "$CTITLE" --format json | jq -c '.'
echo "Reading file YTCL-${NAME}.diff and update object ${NAME}"
cat YTCL-${NAME}.diff | while read URL
do
mgmt_cli --management $MGMTIP -s sid_b.txt set application-site name "YTCL-${NAME}" url-list.add $URL --format json | jq -c '.'
done
else
echo "No changes on YTCL-${NAME}.diff file"
fi
done
echo "MGMT - Publishing Session"
mgmt_cli --management $MGMTIP -s sid_b.txt publish --format json | jq -c '.'
echo "MGMT - Pushing Policy"
mgmt_cli --management $MGMTIP -s sid_b.txt install-policy policy-package "standard" access true threat-prevention false targets.1 "R81.10-StandAlone" --format json | jq -c '.'
echo "MGMT - Logout"
mgmt_cli --management $MGMTIP -s sid_b.txt logout --format json | jq -c '.'
echo "#### End of Execution of ytcl_update.sh at $(date -u) ####"
|
Step by step
- Enable the Functions: “Application Control” and “URL Filtering”.
- Enable the SSL Inspection Functionality “Enable HTTPS Inspection”.
- Create the “YouTube-Domain” object
- Add YouTube domain: \.youtube\.com
- Check the option “URLs are defined as Regular Expression”
- Create the object “Application/Site Group” and name it “YouTubeChannels-AllowList”
- Add the YouTube domain: \.youtube\.com
- Check the option “URLs are defined as Regular Expression”
- Add to the group “YouTubeChannels-AllowList”
- Click on the + button
- Click on the * button
- Select the option “Application/Site...”
- Configure the group using the following pattern
- Name Nomenclature: YTCL-{Channel ID}
- Add all the base URLs you want to allow for the channel:
- Uncheck the option “URLs are defined as Regular Expression”
- Configure access rules for:
- Blocking the QUIC protocol
- Web Filtering access rule with Inline Layer (Application Control & URL Filtering)
- Allow rule for the YouTube Channel List (YouTubeChannels-AllowList)
- Block rule for the YouTube Domain (YouTube-Domain)
- Allow rule for the YouTube Application
- Copy the 03 (three) script files to the Check Point Management Server
- Create the apikey file containing the Check Point Management user API key and the gpaikey file containing the Google API key to query via YouTube Data API v3
- Run the ytcl_watch.sh script
- Check the content of the object “YouTubeChannels-AllowList” and look for Application/Site that start with the prefix “YTCL-“ and save it on the file “YouTubeChannels-AllowList.list”.
- Compares the contents of the file “YouTubeChannels-AllowList.list” with the file “YouTubeChannels-AllowList.list1” to see if any new channels have been added and generates the file “YouTubeChannels-AllowList.diff”
- If the “YouTubeChannels-AllowList.diff” file has any information it will run a new script named “ytcl_diff.sh” to create the YTCL-{Channel_ID}.url files with the URLs of all the public videos of each new chaneel that was added to the “YouTubeChannels-AllowList” object
- Automatic execution of the ytcl_diff.sh script to create the {Channel_ID}.url files, if the “YouTubeChannels-AllowList.diff” file is not empty
- The contents of the “YouTubeChannels-AllowList” file will be moved to the “YouTubeChannels-AllowList1” file to be used as a comparison basis for the next script execution.
- Automatic execution of the ytcl_update.sh script if there are YTCL-{Channel_ID}.url files in the directory to update the objects referring to YouTube Channels with the list of URLs that are not yet registered, in addition to updating the COMMENTS field with the channel title.
- The script moves the YTCL-{Channel_ID}.url files generated in the previous step to the YTCL-{Channel_ID}.bkp-$(date +%Y%m%d%H%M%S) format to serve as a history of which URLs were added at that time
- Automatic execution of the ytcl_diff.shscript
- For each line of the “YouTubeChannels-AllowList.diff” the following actions are performed:
- Parsing to find the YouTube Channel ID, as defined in the creation of objects of the “Application/Site” type following the YTCL-{Channel_ID} rule
- Delete, if any, the contents of the YTCL-{Channel_ID}.url file
- Check if the Channel ID is valid and if the Channel has public video
- Recursively list all the channel's public video URLs and save to {Channel_ID}.url file
- Automatic execution of the ytcl_diff.sh script
- Extracts the YouTube channel ID that was added to the “YouTubeChannels-AllowList” object as per specified naming YTCL-{Channel_ID}
- Query the object of type “Application/Site” added to the group “YouTubeChannels-AllowList” and save the current URL list of each object in a file “YTCL-{Channel_ID}.list”
- Checks the difference between the files “YTCL-{Channel_ID}.url”, generated by the script ytcl_diff.sh and the newly created file “YTCL-{Channel_ID}.list” and saves the additional URLs of the file “YTCL-{Channel_ID}.url” in another “YTCL-{Channel_ID}.diff file
- Updates the “Comments” field of the “YTCL-{Channel_ID}” object with the corresponding YouTube Channel Title
- Updates the list of URLs of the object “YTCL-{Channel_ID}” with the additional URLs saved in the file “YTCL-{Channel_ID}.diff”
- Publish the session
- Install the Policy
- List of files in directory after execution
