How to allow specific YouTube channels

Thiago_Mourao — Wed, 27 Oct 2021 15:10:12 GMT

How to allow specific YouTube channels

Security Engineering Brazil

July, 2021

Versão 02

Author:

Thiago Mourao, SE

Revision:

Contents

How to allow specific YouTube channels

Goal

Warning

Limitations and Requirements

Topology

Scripts

Step by step

Goal

The purpose of the document is to describe the steps necessary to make a granular control of the YouTube channel in Check Point's NGFW, thus being possible to allow or block access to specific YouTube channels from their unique identifier.

Warning

The script used in this Proof of Concept was created by me (Thiago Mourão) using official reference documents from the API Management API Reference v1.8 (link), for more details and updates to the API available through Check Point, please visit the Check Point API Reference (link).

Limitations and Requirements

For this lab, version 1.8 of the Check Point management API was used, available from version R81.10 of the Gaia Operating System. For earlier versions, please refer to the corresponding version reference guide to ensure support for the commands used.
To list videos from a YouTube channel, it was used in YouTube Data API v3 (link) and for that you will need to create your own key for this API.
To do granular control of YouTube video URLs it will be necessary configure the HTTPS inspection (SSL Inspection) on the gateway

Topology

In this lab, we used an R81.10 version of Check Point Gaia defined as Security Gateway and Manager (StandAlone) at the same time and they were virtualized on VMware Workstation 16.

Virtual Environment:

1 x Check Point StandAlone (Gateway/ Manager) R81.10
1 x Microsoft Windows 10

Scripts

ytcl_watch.sh

This script performs the following steps:

Checks whether new YouTube channels have been considered for the “YouTubeChannels-AllowList” object in the format defined as YTCL-Channel_ID
If a new channel has been added to the scripts “ytcl_diff.sh” and “ytcl_update.sh” it should be possible, in case there are no errors and the damaged channels have public videos.

#!/bin/bash -f

source /var/opt/CPshrd-R81.10/tmp/.CPprofile.sh

#Script Directory
EXECDIR="/home/admin/ytcl"
APIKEY=$(cat apikey)
MGMTIP="192.168.157.100"

cd $EXECDIR

echo "#### Executing ytcl_watch.sh at $(date -u) ####"
echo "MGMT - Login"
mgmt_cli login --management $MGMTIP api-key $APIKEY > sid_a.txt

echo "Erasing YouTubeChannels-AllowList.list file"
> YouTubeChannels-AllowList.list

echo "Getting information from YouTubeChannels-AllowList Object and saving on file YouTubeChannels-AllowList.list"
mgmt_cli --management $MGMTIP -s sid_a.txt show application-site-group name "YouTubeChannels-AllowList" --format json | jq '.members[] | select( .name | contains("YTCL"))' | jq '.name' | sort > YouTubeChannels-AllowList.list

echo "MGMT - Logout"
mgmt_cli --management $MGMTIP -s sid_a.txt logout --format json | jq -c '.'

echo "Erasing YouTubeChannels-AllowList.diff file"
> YouTubeChannels-AllowList.diff

echo "Creating new YouTubeChannels-AllowList.diff file"
diff <(sort YouTubeChannels-AllowList.list) <(sort YouTubeChannels-AllowList.list1) | grep "<" | sed 's/< //' > YouTubeChannels-AllowList.diff

echo "Checking YouTubeChannels-AllowList.diff file"
if [[ $(cat YouTubeChannels-AllowList.diff | wc -l) -gt 0 ]]

then
echo "Running script $EXECDIR/ytcl_diff.sh to creating .url files"
sh $EXECDIR/ytcl_diff.sh

else

echo "No changes on YouTubeChannels-AllowList.list file"

echo "Copying file $EXECDIR/YouTubeChannels-AllowList.list to $EXECDIR/YouTubeChannels-AllowList.list1"
cp -f $EXECDIR/YouTubeChannels-AllowList.list $EXECDIR/YouTubeChannels-AllowList.list1

if [[ $(sh -c 'ls *.url | grep YTCL' | wc -l) -gt 0 ]]
then
echo "Running script $EXECDIR/ytcl_update.sh to Updating URLs on Group YouTubeChannels-AllowList"
sh $EXECDIR/ytcl_update.sh
else
echo "No YTCL.url files were found"
fi

echo "Backup last URL files"
sh -c 'ls *.url | grep YTCL' | sed 's/$YTCL.*$\.url/\1/' | while read NAME
do
mv -f ${NAME}.url ${NAME}.bkp-$(date +%Y%m%d%H%M%S)
done

echo "#### End of Execution of ytcl_watch.sh at $(date -u) ####"

ytcl_diff.sh

This script performs the following steps:

Check if the channel ID exists and if it has public videos
If the Channel exists and has public videos, recursively, using a YouTube Data API v3, all URLs of public videos are extracted and saved in a file.

#!/bin/bash -f

source /var/opt/CPshrd-R81.10/tmp/.CPprofile.sh

#Script Directory
EXECDIR="/home/admin/ytcl"

#Google API Variables
GAPIKEY=$(cat gapikey)
CID=""
MR=5
NTP_S=""

echo "#### Executing ytcl_diff.sh at $(date -u) ####"

cat YouTubeChannels-AllowList.diff | sed 's/"YTCL\-$.*$"/\1/' | while read CID
do
if [[ $(curl_cli -s -k "https://www.googleapis.com/youtube/v3/search?key=$GAPIKEY&channelId=$CID&part=snippet,id&order=date&maxResults=1" | jq '.pageInfo.totalResults') -gt 0 ]]
then
echo "Erasing file YTCL-$CID.url"
> YTCL-${CID}.url
curl_cli -k -s "https://www.googleapis.com/youtube/v3/search?key=$GAPIKEY&channelId=$CID&part=snippet,id&order=date&maxResults=$MR" > result-${CID}.txt
while true
do
if [[ $(cat result-${CID}.txt | jq '.' | grep "nextPageToken" | wc -l) -gt 0 ]]
then
echo "nextPageToken ($NTP_S) found for channel $CID"
NTP_S="$(cat result-${CID}.txt | jq '.' | grep nextPageToken | cut -d '"' -f 4)"
echo "Adding URLs to file YTCL-${CID}.urls - nextPageToken will be $NTP_S"
cat result-${CID}.txt | jq '.items[].id.videoId' | grep -v null | sed 's/^\"/\"www\.youtube\.com\/watch\?v\=/g' >> YTCL-${CID}.url
curl_cli -k -s "https://www.googleapis.com/youtube/v3/search?key=$GAPIKEY&channelId=$CID&part=snippet,id&order=date&maxResults=$MR&pageToken=$NTP_S" > result-${CID}.txt
else
echo "nextPageToken not found or last page for channel $CID"
echo "Adding last page of URLs to file YTCL-$CID.urls"
cat result-${CID}.txt | jq '.items[].id.videoId' | grep -v null | sed 's/^\"/\"www\.youtube\.com\/watch\?v\=/g' >> YTCL-${CID}.url
echo "File $CID.url finished"
break
fi
done
else
echo "YouTube Channel $CID - No public videos found or channel ID is wrong"
fi
done

echo "#### End of Execution of ytcl_diff.sh at $(date -u) ####"

ytcl_update.sh

This script performs the following steps:

Updates the corresponding YouTube Channel object with the channel title in the “COMMENTS” field, which can be used as a search parameter in SmartConsole in the object tree
Check which URLs of public videos identified in the previous process have not yet been registered in the corresponding channel object and add them.

#!/bin/bash -f

source /var/opt/CPshrd-R81.10/tmp/.CPprofile.sh

#Script Directory
EXECDIR="/home/admin/ytcl"

#Check Point Management Variables
APIKEY=$(cat apikey)
MGMTIP="192.168.157.100"

#Google API Variables
GAPIKEY=$(cat gapikey)

cd $EXECDIR

echo "#### Executing ytcl_update.sh at $(date -u) ####"

echo "MGMT - Login"
mgmt_cli login --management $MGMTIP api-key $APIKEY > sid_b.txt

sh -c 'ls *.url' |grep YTCL | sed 's/YTCL\-$.*$\.url/\1/' | while read NAME
do
mgmt_cli --management $MGMTIP -s sid_b.txt show application-site name "YTCL-${NAME}" --format json | jq '."url-list"[]' | sort > YTCL-${NAME}.list

echo "Creating new YTCL-${NAME}.diff file"
diff YTCL-${NAME}.url YTCL-${NAME}.list | grep "<" | sed 's/< //' > YTCL-${NAME}.diff

echo "Checking YTCL-${NAME}.diff file"
if [[ $(cat YTCL-${NAME}.diff | wc -l) -gt 0 ]]
then
echo "Updating Channel Title on Object comments"
CTITLE=$(curl_cli -k -s "https://www.googleapis.com/youtube/v3/search?key=$GAPIKEY&channelId=${NAME}&part=snippet,id&order=date&maxResults=1" | jq '.items[].snippet.channelTitle')
echo "mgmt_cli --management $MGMTIP -s sid_b.txt set application-site name \"YTCL-${NAME}\" comments $CTITLE --format json"
mgmt_cli --management $MGMTIP -s sid_b.txt set application-site name "YTCL-${NAME}" comments "$CTITLE" --format json | jq -c '.'

echo "Reading file YTCL-${NAME}.diff and update object ${NAME}"
cat YTCL-${NAME}.diff | while read URL
do
mgmt_cli --management $MGMTIP -s sid_b.txt set application-site name "YTCL-${NAME}" url-list.add $URL --format json | jq -c '.'
done
else
echo "No changes on YTCL-${NAME}.diff file"
fi

done

echo "MGMT - Publishing Session"
mgmt_cli --management $MGMTIP -s sid_b.txt publish --format json | jq -c '.'

echo "MGMT - Pushing Policy"
mgmt_cli --management $MGMTIP -s sid_b.txt install-policy policy-package "standard" access true threat-prevention false targets.1 "R81.10-StandAlone" --format json | jq -c '.'

echo "MGMT - Logout"
mgmt_cli --management $MGMTIP -s sid_b.txt logout --format json | jq -c '.'

echo "#### End of Execution of ytcl_update.sh at $(date -u) ####"

Step by step

Enable the Functions: “Application Control” and “URL Filtering”.
Enable the SSL Inspection Functionality “Enable HTTPS Inspection”.
Create the “YouTube-Domain” object
- Add YouTube domain: \.youtube\.com
- Check the option “URLs are defined as Regular Expression”
Create the object “Application/Site Group” and name it “YouTubeChannels-AllowList”
- Add the YouTube domain: \.youtube\.com
- Check the option “URLs are defined as Regular Expression”
Add to the group “YouTubeChannels-AllowList”
- Click on the + button
- Click on the * button
- Select the option “Application/Site...”
Configure the group using the following pattern
- Name Nomenclature: YTCL-{Channel ID}
- Add all the base URLs you want to allow for the channel:
- Uncheck the option “URLs are defined as Regular Expression”
Configure access rules for:
- Blocking the QUIC protocol
- Web Filtering access rule with Inline Layer (Application Control & URL Filtering)
  - Allow rule for the YouTube Channel List (YouTubeChannels-AllowList)
  - Block rule for the YouTube Domain (YouTube-Domain)
  - Allow rule for the YouTube Application
Copy the 03 (three) script files to the Check Point Management Server
Create the apikey file containing the Check Point Management user API key and the gpaikey file containing the Google API key to query via YouTube Data API v3
Run the ytcl_watch.sh script
1. Check the content of the object “YouTubeChannels-AllowList” and look for Application/Site that start with the prefix “YTCL-“ and save it on the file “YouTubeChannels-AllowList.list”.
2. Compares the contents of the file “YouTubeChannels-AllowList.list” with the file “YouTubeChannels-AllowList.list1” to see if any new channels have been added and generates the file “YouTubeChannels-AllowList.diff”
3. If the “YouTubeChannels-AllowList.diff” file has any information it will run a new script named “ytcl_diff.sh” to create the YTCL-{Channel_ID}.url files with the URLs of all the public videos of each new chaneel that was added to the “YouTubeChannels-AllowList” object
4. Automatic execution of the ytcl_diff.sh script to create the {Channel_ID}.url files, if the “YouTubeChannels-AllowList.diff” file is not empty
5. The contents of the “YouTubeChannels-AllowList” file will be moved to the “YouTubeChannels-AllowList1” file to be used as a comparison basis for the next script execution.
6. Automatic execution of the ytcl_update.sh script if there are YTCL-{Channel_ID}.url files in the directory to update the objects referring to YouTube Channels with the list of URLs that are not yet registered, in addition to updating the COMMENTS field with the channel title.
7. The script moves the YTCL-{Channel_ID}.url files generated in the previous step to the YTCL-{Channel_ID}.bkp-$(date +%Y%m%d%H%M%S) format to serve as a history of which URLs were added at that time
Automatic execution of the ytcl_diff.shscript
1. For each line of the “YouTubeChannels-AllowList.diff” the following actions are performed:
2. Parsing to find the YouTube Channel ID, as defined in the creation of objects of the “Application/Site” type following the YTCL-{Channel_ID} rule
3. Delete, if any, the contents of the YTCL-{Channel_ID}.url file
4. Check if the Channel ID is valid and if the Channel has public video
5. Recursively list all the channel's public video URLs and save to {Channel_ID}.url file
Automatic execution of the ytcl_diff.sh script
1. Extracts the YouTube channel ID that was added to the “YouTubeChannels-AllowList” object as per specified naming YTCL-{Channel_ID}
2. Query the object of type “Application/Site” added to the group “YouTubeChannels-AllowList” and save the current URL list of each object in a file “YTCL-{Channel_ID}.list”
3. Checks the difference between the files “YTCL-{Channel_ID}.url”, generated by the script ytcl_diff.sh and the newly created file “YTCL-{Channel_ID}.list” and saves the additional URLs of the file “YTCL-{Channel_ID}.url” in another “YTCL-{Channel_ID}.diff file
4. Updates the “Comments” field of the “YTCL-{Channel_ID}” object with the corresponding YouTube Channel Title
5. Updates the list of URLs of the object “YTCL-{Channel_ID}” with the additional URLs saved in the file “YTCL-{Channel_ID}.diff”
6. Publish the session
7. Install the Policy
List of files in directory after execution

Re: How to allow specific YouTube channels

DavidMM27 — Thu, 20 Apr 2023 16:44:34 GMT

Hi Thiago_Mourao

Thank you for the useful information, but we have an issue with the step 9, we want to know how to create the apikey and gapikey files and what does that files contain?

Thanks

Re: How to allow specific YouTube channels

PhoneBoy — Fri, 21 Apr 2023 20:34:44 GMT

You have to do it in the Google Developers Console.
See: https://console.developers.google.com/

Re: How to allow specific YouTube channels

laks_Sparrow — Tue, 11 Mar 2025 12:19:01 GMT

The ytcl_watch.sh is not working

topic How to allow specific YouTube channels in API / CLI Discussion

How to allow specific YouTube channels

Re: How to allow specific YouTube channels

Re: How to allow specific YouTube channels

Re: How to allow specific YouTube channels