SEARCH

Enter your search query in the box above ^, or use the forum search tool.

You are not logged in.

#1 2014-11-23 13:58:54

twoion
Emerald Caffeine
From: 星界
Registered: 2012-05-11
Posts: 1,648

[script] get-youtubedl: download recent youtube-dl versions securely

There is a youtube-dl package in jessie and sid, but due to the rather rapid development cycle of youtube-dl, it is almost always obsolete. Downloading youtube-dl using the shell one-liners listed on the upstream site is dangerous, since they do not check the signatures of the downloaded binary and allow broken versions of TLS and SSL to be used when downloading.

This must not be.

So as at a temporary solution, I present a script much in the spirit of update-flashplugin-nonfree et al., that installs/updates to the most recent youtube-dl version but makes sure to only use TLSv1.2 and verify all signatures. It also supports just comparing the local and the remote version. It plays together well with tools like XStow or GNU stow.

The script | Debian package | Repository

Here's the synopsis:

Usage: ./get-youtubedl [-CWah] -c|-i|-u [-t TARGET]
Options:
  -C    Use curl for downloading stuff from the internet,
        this is the default
  -S    Use sudo for installing, this will prompt for the
        sudo password
  -W    Use wget for downloading stuff from the internet
  -a    Try to automatically detect the location of the
        available youtube-dl binary when updating (automates -t).
  -c    Check if there is a more recent version available,
        exits with 0 if the local version is up-to-date and
        1 if the local version is older than the upstream
  -h    Print this message
  -i    Install the most recent version 
  -t    Specify the directory to install youtube-dl to,
        defaults to /usr/local/bin
  -u    Install only if the upstream version is more recent
        than the local version

--
This is totally free software. Use at your own risk.

Last edited by twoion (2015-01-30 19:37:37)


Tannhäuser ~ {www,pkg,ddl}.bunsenlabs.org/{gitlog,repoidx}

Offline

Be excellent to each other!

#2 2014-11-23 20:23:00

cloverskull
#! Junkie
Registered: 2013-10-26
Posts: 377

Re: [script] get-youtubedl: download recent youtube-dl versions securely

Excellent, thank you sir! smile

Offline

#3 2014-11-24 06:50:53

johnraff
nullglob
From: Nagoya, Japan
Registered: 2009-01-07
Posts: 4,148
Website

Re: [script] get-youtubedl: download recent youtube-dl versions securely

Thank you twoion! I will use this very soon.  cool


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
#! forum moderator    BunsenLabs

Offline

#4 2014-12-19 23:42:56

Alad
Software Satan
Registered: 2014-02-20
Posts: 1,512

Re: [script] get-youtubedl: download recent youtube-dl versions securely

Trying this on Salix, but it seems to have no effect:

$ ls -l /usr/local/bin
total 2424
-rwxr-xr-x 1 root root 2475450 Dec 17 22:26 liferea
-rwxr-xr-x 1 root root     488 Dec 17 22:26 liferea-add-feed
math[~]$ bash -x get-youtubedl.sh -S
+ set -e
+ set -f
+ readonly URL=https://yt-dl.org/latest
+ URL=https://yt-dl.org/latest
++ mktemp -d
+ readonly TMPDIR=/tmp/tmp.kQIaag
+ TMPDIR=/tmp/tmp.kQIaag
+ TARGETDIR=/usr/local/bin
+ DOWNLOADCMD=curl
+ MODE=noop
+ EXITC=0
+ USESUDO=0
+ trap cleanup EXIT
+ trap cleanup SIGTERM
+ getopts CSWhict:u OPT
+ case "$OPT" in
+ USESUDO=1
+ getopts CSWhict:u OPT
+ type curl
+ case "$MODE" in
+ exit 0
+ cleanup
+ rm -rf /tmp/tmp.kQIaag
ls -l /usr/local/bin
total 2424
-rwxr-xr-x 1 root root 2475450 Dec 17 22:26 liferea
-rwxr-xr-x 1 root root     488 Dec 17 22:26 liferea-add-feed

edit: OK it seems I'm turning blind, as I've missed the -i option. But:

$ bash -x ./get-youtubedl.sh -i -S
+ set -e
+ set -f
+ readonly URL=https://yt-dl.org/latest
+ URL=https://yt-dl.org/latest
++ mktemp -d
+ readonly TMPDIR=/tmp/tmp.A2wQN3
+ TMPDIR=/tmp/tmp.A2wQN3
+ TARGETDIR=/usr/local/bin
+ DOWNLOADCMD=curl
+ MODE=noop
+ EXITC=0
+ USESUDO=0
+ trap cleanup EXIT
+ trap cleanup SIGTERM
+ getopts CSWhict:u OPT
+ case "$OPT" in
+ MODE=install
+ getopts CSWhict:u OPT
+ case "$OPT" in
+ USESUDO=1
+ getopts CSWhict:u OPT
+ type curl
+ case "$MODE" in
+ download_and_verify /tmp/tmp.A2wQN3/youtube-dl
+ local placein=/tmp/tmp.A2wQN3/youtube-dl
+ exec_download https://yt-dl.org/latest/youtube-dl /tmp/tmp.A2wQN3/youtube-dl
+ [[ curl = wget ]]
+ exec_curl https://yt-dl.org/latest/youtube-dl /tmp/tmp.A2wQN3/youtube-dl
+ local url=https://yt-dl.org/latest/youtube-dl output=/tmp/tmp.A2wQN3/youtube-dl
+ curl --silent --tlsv1.2 --output /tmp/tmp.A2wQN3/youtube-dl https://yt-dl.org/latest/youtube-dl
+ return 0
+ return 0
+ exec_download https://yt-dl.org/latest/youtube-dl.sig /tmp/tmp.A2wQN3/youtube-dl.sig
+ [[ curl = wget ]]
+ exec_curl https://yt-dl.org/latest/youtube-dl.sig /tmp/tmp.A2wQN3/youtube-dl.sig
+ local url=https://yt-dl.org/latest/youtube-dl.sig output=/tmp/tmp.A2wQN3/youtube-dl.sig
+ curl --silent --tlsv1.2 --output /tmp/tmp.A2wQN3/youtube-dl.sig https://yt-dl.org/latest/youtube-dl.sig
+ return 0
+ return 0
+ exec_download https://yt-dl.org/latest/SHA2-512SUMS /tmp/tmp.A2wQN3/SHA2-512SUMS
+ [[ curl = wget ]]
+ exec_curl https://yt-dl.org/latest/SHA2-512SUMS /tmp/tmp.A2wQN3/SHA2-512SUMS
+ local url=https://yt-dl.org/latest/SHA2-512SUMS output=/tmp/tmp.A2wQN3/SHA2-512SUMS
+ curl --silent --tlsv1.2 --output /tmp/tmp.A2wQN3/SHA2-512SUMS https://yt-dl.org/latest/SHA2-512SUMS
+ return 0
+ return 0
+ gpg --verify /tmp/tmp.A2wQN3/youtube-dl.sig /tmp/tmp.A2wQN3/youtube-dl
+ echo 'Couldn'\''t verify signature, youtube-dl might be corrupted or compromised. Abort.'
Couldn't verify signature, youtube-dl might be corrupted or compromised. Abort.
+ return 1
+ cleanup
+ rm -rf /tmp/tmp.A2wQN3

Hmm sad

Last edited by Alad (2014-12-19 23:45:32)

Offline

#5 2014-12-20 10:18:04

twoion
Emerald Caffeine
From: 星界
Registered: 2012-05-11
Posts: 1,648

Re: [script] get-youtubedl: download recent youtube-dl versions securely

The same invocation "get-youtubedl.sh -iS" works fine on my Debian Sid. Are you sure that gpg is installed, as opposed to, say, only the "gpg2" binary? Admittedly, the gpg version that is installed should be auto-detected.
--
Edit: I added a check to the script.

Last edited by twoion (2014-12-20 10:28:16)


Tannhäuser ~ {www,pkg,ddl}.bunsenlabs.org/{gitlog,repoidx}

Offline

#6 2014-12-20 11:01:12

Alad
Software Satan
Registered: 2014-02-20
Posts: 1,512

Re: [script] get-youtubedl: download recent youtube-dl versions securely

Yep, both are installed:

$ gpg --version
gpg (GnuPG) 1.4.17
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: ~/.gnupg
Supported algorithms:
Pubkey: RSA, RSA-E, RSA-S, ELG-E, DSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

$ gpg2 --version
gpg (GnuPG) 2.0.24
libgcrypt 1.5.3
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: ~/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

Offline

#7 2014-12-20 11:42:05

twoion
Emerald Caffeine
From: 星界
Registered: 2012-05-11
Posts: 1,648

Re: [script] get-youtubedl: download recent youtube-dl versions securely

OK, I've added a check for zero-sized or inaccessible files that were downloaded. If the signature verification fails and all permissions are correct/gpg can access all necessary files, then the signature simply doesn't match the downloaded file.

Could you remove the &>/dev/null behind the GPG invocation in line 124 in order to get the actual error that GPG is (possibly) reporting in order to make sure?

I am running this on my Arch system without any problems.


Tannhäuser ~ {www,pkg,ddl}.bunsenlabs.org/{gitlog,repoidx}

Offline

#8 2014-12-20 12:00:39

Alad
Software Satan
Registered: 2014-02-20
Posts: 1,512

Re: [script] get-youtubedl: download recent youtube-dl versions securely

After removing the /dev/null (also on line 76), I get this:

+ gpg --verify /tmp/tmp.lDJP23/youtube-dl.sig /tmp/tmp.lDJP23/youtube-dl
gpg: Signature made Wed 17 Dec 2014 11:40:26 AM CET using RSA key ID A4826A18
gpg: Can't check signature: public key not found

So the script didn't import the key first tongue (slap me if you've mentioned this somewhere, but I don't see it right now)

Last edited by Alad (2014-12-20 12:01:53)

Offline

#9 2014-12-20 12:31:56

twoion
Emerald Caffeine
From: 星界
Registered: 2012-05-11
Posts: 1,648

Re: [script] get-youtubedl: download recent youtube-dl versions securely

Alad wrote:

After removing the /dev/null (also on line 76), I get this:

+ gpg --verify /tmp/tmp.lDJP23/youtube-dl.sig /tmp/tmp.lDJP23/youtube-dl
gpg: Signature made Wed 17 Dec 2014 11:40:26 AM CET using RSA key ID A4826A18
gpg: Can't check signature: public key not found

So the script didn't import the key first tongue (slap me if you've mentioned this somewhere, but I don't see it right now)

Oooopps I totally forgot to mention that big_smile I added a key import routine to the script smile


Tannhäuser ~ {www,pkg,ddl}.bunsenlabs.org/{gitlog,repoidx}

Offline

#10 2014-12-20 13:37:45

Alad
Software Satan
Registered: 2014-02-20
Posts: 1,512

Re: [script] get-youtubedl: download recent youtube-dl versions securely

Thanks, works perfectly now.  big_smile

Offline

#11 2014-12-20 17:10:31

paxmark1
#! Junkie
From: Winnipeg, MB Canada
Registered: 2009-07-08
Posts: 263

Re: [script] get-youtubedl: download recent youtube-dl versions securely

As interest might rise soon when testing changes, I have a couple of questions.

1.  I tried putting it into ~/bin and chmod +x   I had no luck.  I read more of above and put it into /usr/local/bin/ where the smxi's reside with apparent success.  Is /usr /local/bin/  the most appropriate place or was I doing something incorrect when I place it in ~/bin?

2.  I skimmed a little fast  instead of reading deeply. My first thought and impressions was that this was a wrapper to be used each time in place of youtube-dl instead of an occasional tool (weekly, monthly) to be used to keep youtube-dl at the leading edge.    Correct me if I am wrong. 

Sid on siduction lxqt(5) and openbox appears to work

PGP signature successfully verified
SHA512 checksum successfully verified
rauneseee:~/bin$ getyoutube-dl -c
Local version: 2014.12.17.2
Upstream version: 2014.12.17.2
The local version is up-to-date.

Thanks


Jean Vanier wrote "Being Human" and "A Short History of Progress" by Ronald Wright.  Gotta love the Massey Lectures.

Offline

#12 2014-12-20 17:51:17

twoion
Emerald Caffeine
From: 星界
Registered: 2012-05-11
Posts: 1,648

Re: [script] get-youtubedl: download recent youtube-dl versions securely

paxmark1 wrote:

1.  I tried putting it into ~/bin and chmod +x   I had no luck.  I read more of above and put it into /usr/local/bin/ where the smxi's reside with apparent success.  Is /usr /local/bin/  the most appropriate place or was I doing something incorrect when I place it in ~/bin?

I infer that you mean you have placed the youtube-dl binary in ~/bin. Firstly, you do not need to use chmod since the installation routine in the script will already set an appropriate mode (755 if installed with sudo, 700 otherwise). Secondly, ~/bin needs to be in your $PATH if you just want to run 'youtube-dl' on the command line. In fact, when comparing remote and local versions, the script also just calls 'youtube-dl' and thus will use the first 'youtube-dl' binary it finds in accordance with the path list in your $PATH variable. For example, if you have PATH=$HOME/bin:/usr/local/bin and there is a youtube-dl binary in $HOME/bin and /usr/local/bin, the one in $HOME/bin will be 'active' since that's the order in which the shell will look up the binary to run

2.  I skimmed a little fast  instead of reading deeply. My first thought and impressions was that this was a wrapper to be used each time in place of youtube-dl instead of an occasional tool (weekly, monthly) to be used to keep youtube-dl at the leading edge.    Correct me if I am wrong.

No, it's a kind of "installation and/or updating script" that is to be used to keep youtube-dl up-to-date. The reason it exists is that the version in the Debian repos is almost always behind the upstream version (which gets updated every other day) and the download instructions on the project website mainly consist of a one-liner that goes 'wget -O- https://FOO | sudo bash', relying only on SSL (they seem to have disabled SSLv2,v3 in the meantime, but TLSv1.0,v1.1 are still active [TLSv1.2] is what should be used) for trust and integrity while not checking the signature. And One Simply Does Not Execute stuff from the internet directly as root. This script forces TLSv1.2 for CURL (not wget, since WGET CRASHES when I do that on my system [wtf?]), and verifies the secure PGP signature and additionally checks the SHA2 checksum. The installation will fail if any of the trust paths (TLS, PGP, SHA2) has been invalidated. It's a script to securely but easily keep ytdl up-to-date. It can be run from cron, for example.


Tannhäuser ~ {www,pkg,ddl}.bunsenlabs.org/{gitlog,repoidx}

Offline

#13 2014-12-20 17:52:56

twoion
Emerald Caffeine
From: 星界
Registered: 2012-05-11
Posts: 1,648

Re: [script] get-youtubedl: download recent youtube-dl versions securely

Alad wrote:

Thanks, works perfectly now.  big_smile

No, thank *you* for testing!


Tannhäuser ~ {www,pkg,ddl}.bunsenlabs.org/{gitlog,repoidx}

Offline

#14 2014-12-23 15:19:05

twoion
Emerald Caffeine
From: 星界
Registered: 2012-05-11
Posts: 1,648

Re: [script] get-youtubedl: download recent youtube-dl versions securely

I have relased a Debian package for easy installation and moved the script to its own repository.

Bash script | Debian package | Repository
--
Edit: Forgot to make the download location public, now it is smile

Last edited by twoion (2014-12-23 15:29:07)


Tannhäuser ~ {www,pkg,ddl}.bunsenlabs.org/{gitlog,repoidx}

Offline

#15 2015-01-30 19:38:07

twoion
Emerald Caffeine
From: 星界
Registered: 2012-05-11
Posts: 1,648

Re: [script] get-youtubedl: download recent youtube-dl versions securely

Version 1.2 has been released, adding the -a option.

Bash script | Debian package | Repository

The synposis in the OP has been updated accordingly.


Tannhäuser ~ {www,pkg,ddl}.bunsenlabs.org/{gitlog,repoidx}

Offline

#16 2015-02-16 22:46:49

twoion
Emerald Caffeine
From: 星界
Registered: 2012-05-11
Posts: 1,648

Re: [script] get-youtubedl: download recent youtube-dl versions securely

Version 1.3 has been released, the script will now abort when an invalid CLI option has been specified - like most other programs do. It now also does install the README into /usr/share/docs like most other Debian packages.

Bash script | Debian package | Repository


Tannhäuser ~ {www,pkg,ddl}.bunsenlabs.org/{gitlog,repoidx}

Offline

#17 2015-10-02 12:31:33

daggoth
Member
From: New Zealand
Registered: 2012-01-18
Posts: 39

Re: [script] get-youtubedl: download recent youtube-dl versions securely

Thanks for the nice script twoion - it worked okay here without trouble :-)

Offline

Board footer

Powered by FluxBB

Copyright © 2012 CrunchBang Linux.
Proudly powered by Debian. Hosted by Linode.
Debian is a registered trademark of Software in the Public Interest, Inc.
Server: acrobat

Debian Logo