{"id":270,"date":"2021-10-27T19:00:37","date_gmt":"2021-10-27T18:00:37","guid":{"rendered":"https:\/\/asdfghjkl.me.uk\/blog\/?p=270"},"modified":"2021-10-27T19:02:04","modified_gmt":"2021-10-27T18:02:04","slug":"archive-youtube","status":"publish","type":"post","link":"https:\/\/asdfghjkl.me.uk\/blog\/archive-youtube\/","title":{"rendered":"Archiving everything I like with youtube-dl"},"content":{"rendered":"\n

Continuing on the theme of \"link rot bad, hard drives cheap\", a year or so ago I started archiving videos I'd liked or saved to YouTube playlists. You can do this manually without too much trouble but I chucked it in a shell script to run regularly, keeping as much metadata as possible. Here it is!<\/p>\n\n\n

#!\/bin\/bash<\/span>\n<\/span><\/span><\/span>\n<\/span><\/span># Archive youtube videos from a list of channels\/playlists, in up to selected quality,<\/span>\n<\/span><\/span># with formatted filenames and all available metadata in sidecar files.<\/span>\n<\/span><\/span>#<\/span>\n<\/span><\/span># Note: this probably relies on having an up-to-date youtube-dl, so we run<\/span>\n<\/span><\/span># youtube-dl -U in the root crontab an hour before this script runs<\/span>\n<\/span><\/span>\n<\/span><\/span># Settings<\/span>\n<\/span><\/span>quality='bestvideo[height<=?1080]+bestaudio\/best[height<=?1080]'<\/span>\n<\/span><\/span># If we ever get infinite hard drive space:<\/span>\n<\/span><\/span>#quality='bestvideo+bestaudio\/best'<\/span>\n<\/span><\/span># Batch file of URLs to download<\/span>\n<\/span><\/span>batch_file='youtube-list.txt'<\/span>\n<\/span><\/span># File to pull youtube cookies from (for private videos and liked playlist)<\/span>\n<\/span><\/span>cookies_file='youtube-cookies.txt'<\/span>\n<\/span><\/span># Don't download anything absurdly-sized at all (if prefer to download but in worse quality,<\/span>\n<\/span><\/span># add to quality definition instead like [height<=?1080][filesize<10G]<\/span>\n<\/span><\/span>max_filesize='10G'<\/span>\n<\/span><\/span># Clone current useragent (that account is logged in as)<\/span>\n<\/span><\/span>user_agent='Mozilla\/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/83.0.4103.116 Safari\/537.36'<\/span>\n<\/span><\/span># Bind to different IP in case of geo-blocks<\/span>\n<\/span><\/span># Country 1<\/span>\n<\/span><\/span>source_IP='111.111.111.111'<\/span>\n<\/span><\/span># Country 2<\/span>\n<\/span><\/span>#source_IP='222.222.222.222'<\/span>\n<\/span><\/span># ipv6 1 (etc)<\/span>\n<\/span><\/span>#source_IP='2000:aaaa:b:cccc::'<\/span>\n<\/span><\/span># Limit download rate and sleep a random number of seconds between downloads to avoid IP blocks<\/span>\n<\/span><\/span>rate_limit='5M'<\/span>\n<\/span><\/span>sleep_min='10'<\/span>\n<\/span><\/span>sleep_max='15'<\/span>\n<\/span><\/span># Set folder and filename format, and an archive file to avoid redownloading completed videos<\/span>\n<\/span><\/span>filename_format='youtube\/%(playlist)s\/%(playlist_index)05d - %(title)s - %(id)s - %(upload_date)s.%(ext)s'<\/span>\n<\/span><\/span>archive_file='youtube\/youtube-dl-downloaded.txt'<\/span>\n<\/span><\/span>\n<\/span><\/span># Change to directory this script is in (for cron etc)<\/span>\n<\/span><\/span>cd<\/span> $(dirname $0<\/span>) || { echo<\/span> 'Failed to change directory, giving up'<\/span>; exit<\/span> 1; }                              \n<\/span><\/span>\n<\/span><\/span># Explanations<\/span>\n<\/span><\/span>#-sv: simulate verbose for testing<\/span>\n<\/span><\/span>#--playlist-items 1-3: first few only for testing<\/span>\n<\/span><\/span>#--restrict-filenames: replace special characters in case need to transfer to Windows etc<\/span>\n<\/span><\/span>#--no-overwrites: do not overwrite existing files<\/span>\n<\/span><\/span>#--continue: resume partially downloaded files<\/span>\n<\/span><\/span>#--ignore-errors: continue even if a video is unavailable (taken down etc)<\/span>\n<\/span><\/span>#--ignore-config: don't read usual config files<\/span>\n<\/span><\/span>#--download-archive $archive_file: use an archive file to avoid redownloading already-downloaded videos<\/span>\n<\/span><\/span>#--yes-playlist: download the whole playlist, in case we pass a video+playlist link<\/span>\n<\/span><\/span>#--playlist-reverse: may be necessary if index starts from most recent addition?<\/span>\n<\/span><\/span>#--write-description: write video description to a .description file<\/span>\n<\/span><\/span>#--write-info-json: write video metadata to a .info.json file<\/span>\n<\/span><\/span>#--write-annotations: write annotations to a .annotations.xml file, why not<\/span>\n<\/span><\/span>#--write-thumbnail: write thumbnail image to disk<\/span>\n<\/span><\/span>#--write-sub: write subtitles (but not autogenerated)<\/span>\n<\/span><\/span>#--embed-subs: also add them to the video file, why not<\/span>\n<\/span><\/span>#--add-metadata: add metadata to video file<\/span>\n<\/span><\/span>\n<\/span><\/span># Use --cookies to temporarily pass cookies (note must be in UNIX newline format, use notepad++ to convert)<\/span>\n<\/span><\/span># fix youtube-dl not working with cookies in python2<\/span>\n<\/span><\/span># https:\/\/github.com\/ytdl-org\/youtube-dl\/issues\/28640<\/span>\n<\/span><\/span>python3 \/usr\/bin\/youtube-dl \\\n<\/span><\/span>--cookies \"$cookies_file<\/span>\"<\/span> \\\n<\/span><\/span>--batch-file \"$batch_file<\/span>\"<\/span> \\\n<\/span><\/span>--output \"$filename_format<\/span>\"<\/span> \\\n<\/span><\/span>--format \"$quality<\/span>\"<\/span> \\\n<\/span><\/span>--user-agent \"$user_agent<\/span>\"<\/span> \\\n<\/span><\/span>--source<\/span>-address \"$source_IP<\/span>\"<\/span> \\\n<\/span><\/span>--max-filesize \"$max_filesize<\/span>\"<\/span> \\\n<\/span><\/span>--limit<\/span>-rate \"$rate_limit<\/span>\"<\/span> \\\n<\/span><\/span>--sleep-interval \"$sleep_min<\/span>\"<\/span> \\\n<\/span><\/span>--max-sleep-interval \"$sleep_max<\/span>\"<\/span> \\\n<\/span><\/span>--restrict-filenames \\\n<\/span><\/span>--no-overwrites \\\n<\/span><\/span>--no-warnings \\\n<\/span><\/span>--continue<\/span> \\\n<\/span><\/span>--ignore-errors \\\n<\/span><\/span>--ignore-config \\\n<\/span><\/span>--download-archive \"$archive_file<\/span>\"<\/span> \\\n<\/span><\/span>--yes-playlist \\\n<\/span><\/span>--playlist-reverse \\\n<\/span><\/span>--write-description \\\n<\/span><\/span>--write-info-json \\\n<\/span><\/span>--write-annotations \\\n<\/span><\/span>--write-thumbnail \\\n<\/span><\/span>--write-sub \\\n<\/span><\/span>--sub-lang en \\\n<\/span><\/span>--embed-subs \\\n<\/span><\/span>--add-metadata\n<\/span><\/span>\n<\/span><\/span><\/code><\/span>Code language:<\/span> Bash<\/span> (<\/span>bash<\/span>)<\/span><\/small><\/pre>\n\n\n
You'll need the wonderful youtube-dl<\/a> to run this. Should be fairly self-explanatory, but there's a few bits I find especially useful.<\/p>\n\n\n\n
I limit video quality to the best up-to-1080p possible, since 4K videos can be huge and I'm not fussed for an archive. I also put a hard limit on filesize to avoid downloading any 10-hour videos, but you have the option to get them in lower quality instead. I keep the URLs to download in a separate file: these can be individual videos, entire channels or playlists, one on each line.<\/p>\n\n\n\n
You can make your own playlists unlisted if you don't want them public but still want to be able to download them with this script. Unfortunately there is one case where this doesn't work \u2013 your liked videos playlist is always private and can't be changed. youtube-dl does let you pass in the username and password to your Google account but I find this rarely works, so instead you can export your YouTube cookies (using something like this extension<\/a> on a YouTube page), dump them in a .txt file and point youtube-dl to them. It's probably sensible to clone your browser's useragent too, and set some rate limits to not abuse their hospitality too much.<\/p>\n\n\n\n
Since some videos will inevitably be geo-restricted and I have a few IPs pointing to my box that geolocate to different countries, I'll occasionally let it do a run from somewhere else to sweep up any videos that might have been missed.<\/p>\n\n\n\n
Although I save metadata anyway, I try to make the output format descriptive enough that I could live without it. I save each video to a folder named for its playlist\/channel, and name the video with its position in the playlist, title, video ID and upload date. Reversing the playlist order means the position index starts from the first video added to the playlist \u2013 otherwise when more videos are added, the latest becomes the new number 1 and your index becomes useless.<\/p>\n\n\n\n
Next post: doing something with them!<\/p>\n","protected":false},"excerpt":{"rendered":"
Continuing on the theme of \"link rot bad, hard drives cheap\", a year or so ago I started archiving videos I'd liked or saved to YouTube playlists. You can do this manually without too much trouble but I chucked it in a shell script to run regularly, keeping as much metadata as possible. Here it […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[10,6],"tags":[],"_links":{"self":[{"href":"https:\/\/asdfghjkl.me.uk\/blog\/wp-json\/wp\/v2\/posts\/270"}],"collection":[{"href":"https:\/\/asdfghjkl.me.uk\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/asdfghjkl.me.uk\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/asdfghjkl.me.uk\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/asdfghjkl.me.uk\/blog\/wp-json\/wp\/v2\/comments?post=270"}],"version-history":[{"count":4,"href":"https:\/\/asdfghjkl.me.uk\/blog\/wp-json\/wp\/v2\/posts\/270\/revisions"}],"predecessor-version":[{"id":336,"href":"https:\/\/asdfghjkl.me.uk\/blog\/wp-json\/wp\/v2\/posts\/270\/revisions\/336"}],"wp:attachment":[{"href":"https:\/\/asdfghjkl.me.uk\/blog\/wp-json\/wp\/v2\/media?parent=270"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/asdfghjkl.me.uk\/blog\/wp-json\/wp\/v2\/categories?post=270"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/asdfghjkl.me.uk\/blog\/wp-json\/wp\/v2\/tags?post=270"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}