ydown-0.6.5 link downloader / download management

[plinej]

I created this yad script from an idea started by @stemsee at viewtopic.php?t=8467

requirements:
yad, wget & curl
xsel -or- xclip

Once you start this from the menu it will watch for web links from your clipboard and you'll get a popup if you'd like to download the link. I've added support for wetransfer and workupload links in this version 0.2. Also, if you have youtube-dl or yt-dlp installed you can download from youtube links.

0.4.1 - Added ability to edit file name with workupload. Added support for mediafire and google drive. For the best support of google drive install gdown https://github.com/wkentaro/gdown . Added better support for mega.nz with megatools from https://megatools.megous.com/ . Added support for instagram, facebook and twitter video with youtube-dl (or yt-dlp).

0.4.2 - Added more options to preferences, including buttons to install or guide you to the web page to install gdown & megatools. Added script to download yt-dlp if you don't have youtube-dl or yt-dlp installed and click ok. Possible support for sourceforge (seems to work in my minimal testing).

0.4.3 - If your download ends up being a html file and you have yt-dlp or youtube-dl installed the script will retry downloading using that instead. I've noticed it seems to work on alot of video sites. I've also edited the scripts to be a bit less puppy specific so it should work on other distros.

0.4.4 - Added --no-check-certificate to wget options per @Jasper's request. Also added the ability to download video from crowdcast. From Chrome you would need to click inspect -> Element -> copy element (the link that shows the m3u8 link. You'll need yt-dlp or youtube-dl for crowdcast downloads to work. I'm sure you could find the m3u8 link with firefox as well but I haven't tried yet.

0.4.5 - Added support for soundcloud with yt-dlp or youtube-dl

0.4.6 - Added mixcloud support with yt-dlp or youtube-dl. Added extra options when yt-dlp or youtube-dl is the downloading tool (pick the audio and/or video stream(s) to download).

0.4.7 - bugfixes and added yt-dlp update option to preferences

0.4.8 - bugfixes and changed main script to use either wget or curl so only one is required.

0.4.9 - code clean up

0.5.0 - some minor fixes and added @stemsee's gettext/trans function.

0.5.1 - added better support for mediafire and wetransfer

0.5.2 - added support for gofile and if wget is installed then the builtin scripts will update when certain sites are downloaded from

0.5.3 - minor bugfixes

0.5.4 - bugfixes...

0.5.5 - improved downloading video from facebook and some other minor tweaks

0.5.6 - Modified the preferences options

0.5.7 - sped up and improved obtaining link file name

0.5.8 - fixed some bugs, modified code to use yad instead of a terminal (i.e. using yad for a tail viewer or download progress bar with code borrowed from yad-wget & youtube-get3)

0.5.9 - bugfixes, added support for more file hosting sites, improved/simplified the preferences gui.

0.6.0 - bugfixes and added more options in the preferences section

0.6.1 - bugfix

0.6.2 - bugfix

0.6.3 - cleaned up some code and some minor improvements

0.6.4 - bugfixes and minor improvements

0.6.5 - minor improvements

ydown-0.6.5.pet

(17.44 KiB) Downloaded 81 times

For better support of mega.nz install megatools. This was compiled in vanilladpup 10 64bit:

megatools-1.11.1-x86_64.pet

(57.57 KiB) Downloaded 96 times

Source code of megatools:

megatools-1.11.1.20230212.tar.xz

(82.43 KiB) Downloaded 53 times

[stemsee]

if trans script is installed then basic translation of selected unfiltered text could be added with something like

Code: Select all

LANG='English en'
echo -e "$($CLIP)" | trans -e google -b -tl $(echo "$LANG" | awk '{print $NF}') | yad --text-info --tail --listen --wrap &

Of course the language code could be a variable set from preferences....just a suggestion.

[plinej]

@stemsee, not sure what that does. All I get is:

(process:336427): Gtk-WARNING **: 06:45:19.771: Locale not supported by C library.
Using the fallback 'C' locale.
Fontconfig warning: ignoring English en: not a valid language tag

[stemsee]

this is the 'trans' script, decompress to /usr/local/bin,

trans.zip

(41.23 KiB) Downloaded 55 times

[keniv]

@plinej
I downloaded and installed ydown-0.2.pet in F96-CE_4. I then tried to download Brave portable from here viewtopic.php?t=6150. I used the link that was provided in the first post i.e.
https://mega.nz/folder/CPxnQQIJ#9jWxpqpaKUWCFjA-ayteQQ
The output I got from the terminal is shown below.

Code: Select all

--2023-04-22 11:38:58--  https://mega.nz/folder/CPxnQQIJ
Resolving mega.nz (mega.nz)... 31.216.144.5, 31.216.145.5, 2a0b:e46:1:145::5, ...
Connecting to mega.nz (mega.nz)|31.216.144.5|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2339 (2.3K) [text/html]
Saving to: ‘CPxnQQIJ#9jWxpqpaKUWCFjA-ayteQQ’

     0K ..                                                    100%  438M=0s

2023-04-22 11:38:58 (438 MB/s) - ‘CPxnQQIJ#9jWxpqpaKUWCFjA-ayteQQ’ saved [2339/2339]

In my "Downloads" directory I found what looked to be a file with the name ‘CPxnQQIJ#9jWxpqpaKUWCFjA-ayteQQ’ which is the same name as that shown in the code above. This download took place almost instantaneously. As I was not sure what had been downloaded I deleted it.
Can you tell me if the process shown above is what I should expect. Is seeing a name which has nothing to do with the name of the item you thought you had downloaded normal using this .pet?

Regards,

Ken.

[stemsee]

@plinej
I downloaded and installed ydown-0.2.pet in F96-CE_4. I then tried to download Brave portable from here viewtopic.php?t=6150. I used the link that was provided in the first post i.e.
https://mega.nz/folder/CPxnQQIJ#9jWxpqpaKUWCFjA-ayteQQ
The output I got from the terminal is shown below.

Code: Select all

--2023-04-22 11:38:58--  https://mega.nz/folder/CPxnQQIJ
Resolving mega.nz (mega.nz)... 31.216.144.5, 31.216.145.5, 2a0b:e46:1:145::5, ...
Connecting to mega.nz (mega.nz)|31.216.144.5|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2339 (2.3K) [text/html]
Saving to: ‘CPxnQQIJ#9jWxpqpaKUWCFjA-ayteQQ’

     0K ..                                                    100%  438M=0s

2023-04-22 11:38:58 (438 MB/s) - ‘CPxnQQIJ#9jWxpqpaKUWCFjA-ayteQQ’ saved [2339/2339]

In my "Downloads" directory I found what looked to be a file with the name ‘CPxnQQIJ#9jWxpqpaKUWCFjA-ayteQQ’ which is the same name as that shown in the code above. This download took place almost instantaneously. As I was not sure what had been downloaded I deleted it.
Can you tell me if the process shown above is what I should expect. Is seeing a name which has nothing to do with the name of the item you thought you had downloaded normal using this .pet?

Regards,

Ken.

The link is to an online folder on the mega website. I could only download the app using mega website options. Otherwise just a html document is downloaded with the contents

Code: Select all

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta property="og:title" content="118.9 MB folder on MEGA" />
<meta property="og:description" content="2 files" />
<meta property="og:url" content="https://mega.nz/folder/CPxnQQIJ" />
<meta property="og:image" content="https://mega.nz/rich-folder.png" />
<meta property="twitter:card" content="summary" />
<meta property="twitter:title" content="118.9 MB folder on MEGA" />
<meta property="twitter:description" content="2 files" />
<meta property="twitter:url" content="https://mega.nz/folder/CPxnQQIJ" />
<meta property="twitter:image" content="https://mega.nz/rich-folder.png" />

<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta http-equiv="Content-Type" content="text/html, charset=UTF-8" />
<meta name="referrer" content="strict-origin-when-cross-origin" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0"/>
<meta name="google" value="notranslate" />
<link rel="icon" href="/favicon.ico?v=3" type="image/x-icon" />
<link rel="chrome-webstore-item" href="https://chrome.google.com/webstore/detail/bigefpfhnfcobdlfbedofhhaibnlghod" />
<meta name="apple-itunes-app" content="app-id=706857885"/>
</head>
<body id="bodyel" class="theme-light not-logged">
<script type="text/javascript" src="/secureboot.js?r=1681952791" charset="utf-8"></script>
<div id="overlay" style="display:none; width:100%; height:100%; position:absolute; left:0px; right:0px; cursor:wait; z-index:999999;"></div>
<div class="download-overlay" id="download_overlay" style="display:none;"></div>
<div class="nav-overlay hidden"></div>
<div id="pageholder" style="display:none;"></div>
<div id="startholder" class="fmholder"></div>
<div id="fmholder" class="fmholder" style="display:none;"></div>
<a href="" download="filename" id="dllink" style="display:none;"></a>
<div style="position:absolute;" id="startswfdiv"></div>
<div id="tooltip"></div>
<div style="position:absolute; left:0px; width:100%; top:0px; height:60px; z-index:99; display:none;" id="topmenufm">
</div>
<textarea id="chromeclipboard" title="copy" style="position:absolute;top:-100%;left:-100%;opacity:0" readonly></textarea>
<iframe id="i-ping" style="position:absolute;top:-100%;left:-100%;opacity:0;display:none" src="about:blank"></iframe>
</body>
</html>

[stemsee]

this code seems to work well .... maybe more regex on grep.

Code: Select all

	yad_clip() {
	
	if [[ "`echo $clip | grep -iEv '^http|ftp|www'`" ]]; then
		LANG='English en'
		echo -e "$clip" | trans -e google -b -tl $(echo "$LANG" | awk '{print $NF}') | yad --geometry=200x200-200-100 --text-info --tail --undecorated --no-buttons --skip-taskbar --fore=yellow --back=darkblue --wrap --timeout 33 &
	fi

and small change to grep filter '^

Code: Select all

if [[ "`echo $clip | grep -iE '^http|ftp|www'`" != "" ]]; then

for the loop use tail -f instead of tail -n 1

Code: Select all

 while [[ "$run" = true ]]; do
	sleep 1
	newclip=`get_clipboard 2>/dev/null | tail -f`
	if [ "$newclip" != "" ] && [ "$newclip" != "$clip" ]; then
		clip="$newclip"
		yad_clip
	fi
done

[plinej]

@keniv, Certain sites like Mega, Google Drive are going to need scripts since wget won't resolve the actual link to the file. There are extra scripts included for wetransfer and workupload that you could look at and maybe get something working for Mega. If anyone does, I'll be more than happy to add the scripts to support other sites.

[plinej]

I updated to version 0.3. This version will give you a warning message when your download is an ASCII HTML file. If anyone encounters such an error and has a script to download from the site in question that uses wget or curl let me know and I'll add your script to the package. This script is just the start of an idea and will probably require lots of updates to support many different sites. I'm not sure how much time I want to put in to that since I can't see this replacing a good download manager but who knows...

[keniv]

@keniv, Certain sites like Mega, Google Drive are going to need scripts since wget won't resolve the actual link to the file. There are extra scripts included for wetransfer and workupload that you could look at and maybe get something working for Mega. If anyone does, I'll be more than happy to add the scripts to support other sites.

Thanks for your reply and the explanation of the experience I had trying to download from Mega. I'm afraid I have little knowledge of scripting and so will have to rely on others to do this. The reason I thought I would try your .pet was that I've had trouble in the past, with slow downloads, when trying to download large files like isos. I had read suggestions that uget or wget could be used to help with this problem.

This version will give you a warning message when your download is an ASCII HTML file.

I've removed v 0.2 and replaced it with v 0.3 as I think it would have been useful to me to know what I was about to download.

Thanks again,

Ken.

[plinej]

@keniv, version 0.3 will let you know that the downloaded file is HTML after the download not before. I could add code to delete that file but it may be what someone was trying to download. You'll have to delete the file yourself afterwards. The problem lies with sites that redirect but still require cookies, certificates, captchas, etc. Unfortunately, all sites are different so there's not going to be an across the board option for everything. Sites like this board just have links that wget or curl don't have problems with. When user interaction is required to get the direct link this script will never work and will just end up downloading HTML information instead of the intended file.

[plinej]

Posted 0.3a. No real change besides a script that should support mega.nz downloads.

[keniv]

@keniv, version 0.3 will let you know that the downloaded file is HTML after the download not before.

Thanks for that it makes things clearer.

Posted 0.3a. No real change besides a script that should support mega.nz downloads.

As I had v 0.3 installed I thought I'd try to download v 0.3a. I realise this is a tiny download but I thought it was worthwhile just for the test. I used copy link location. This did download to my downloads directory with the correct file name.
I downloaded Brave portable using a browser in the normal way when I had the problem downloading it from https://mega.nz/folder/CPxnQQIJ#9jWxpqpaKUWCFjA-ayteQQ with your .pet. I'll remove v 0.3 and install v 0.3a and try downloading this again and report back.

Regards,

Ken.

Edit:

I'll remove v 0.3 and install v 0.3a and try downloading this again and report back.

I've just tried this and it appears to have worked having downloaded both Brave-portable64.tar.xz and the md5 checksum which checks out. However, the download did not seem to be a lot faster than via a browser although this is purely subjective on my part as I did not actually measure the the time that each download took. Should I expect the wget download to be much faster than a browser download. I do think this is a good edition to your .pet as some devs seem to store stuff for download on Mega. I've read there are sometimes problems downloading from this site with a browser which I think can be browser dependent.

[plinej]

Updated to 0.4.

There is now a checkbox on the download link popup to specify file name. Click the check box and you'll get a popup with the file name the script came up with and then you can edit it. This function is not currently supported on mega.nz or workupload site downloads.

I've also cleaned up some minor things that were bugging me...

[keniv]

Updated to 0.4.

There is now a checkbox on the download link popup to specify file name. Click the check box and you'll get a popup with the file name the script came up with and then you can edit it. This function is not currently supported on mega.nz or workupload site downloads.

I've also cleaned up some minor things that were bugging me...

I've now installed v 0.4.

if you have youtube-dl or yt-dlp installed you can download from youtube links.

I tried downloading from a youtube link. The output from the terminal is shown bellow.

Code: Select all

[generic] Extracting URL: www.youtube.com/watch?v=OlNpTjepfb8
WARNING: [generic] The url doesn't specify the protocol, trying with http
[youtube] Extracting URL: http://www.youtube.com/watch?v=OlNpTjepfb8
[youtube] OlNpTjepfb8: Downloading webpage
[youtube] OlNpTjepfb8: Downloading android player API JSON
[youtube] OlNpTjepfb8: Downloading MPD manifest
[info] OlNpTjepfb8: Downloading 1 format(s): 137+140
[download] Destination: How To Fix Noisy Laptop Fan TOSHIBA L750-1V2 l TOSHIBA PSK2YE Notebook Cleaning Fan Cooler [OlNpTjepfb8].f137.mp4
[download] 100% of  299.51MiB in 00:02:05 at 2.38MiB/s      
[download] Destination: How To Fix Noisy Laptop Fan TOSHIBA L750-1V2 l TOSHIBA PSK2YE Notebook Cleaning Fan Cooler [OlNpTjepfb8].f140.m4a
[download] 100% of   11.71MiB in 00:00:06 at 1.82MiB/s   
[Merger] Merging formats into "How To Fix Noisy Laptop Fan TOSHIBA L750-1V2 l TOSHIBA PSK2YE Notebook Cleaning Fan Cooler [OlNpTjepfb8].mp4"
Deleting original file How To Fix Noisy Laptop Fan TOSHIBA L750-1V2 l TOSHIBA PSK2YE Notebook Cleaning Fan Cooler [OlNpTjepfb8].f140.m4a (pass -k to keep)
Deleting original file How To Fix Noisy Laptop Fan TOSHIBA L750-1V2 l TOSHIBA PSK2YE Notebook Cleaning Fan Cooler [OlNpTjepfb8].f137.mp4 (pass -k to keep)

I'm not sure why I'm downloading these.

Code: Select all

[youtube] OlNpTjepfb8: Downloading webpage
[youtube] OlNpTjepfb8: Downloading android player API JSON
[youtube] OlNpTjepfb8: Downloading MPD manifest

Can you tell me why the above is being downloaded?

I seem also to be downloading two formats.

Code: Select all

[download] Destination: How To Fix Noisy Laptop Fan TOSHIBA L750-1V2 l TOSHIBA PSK2YE Notebook Cleaning Fan Cooler [OlNpTjepfb8].f137.mp4
[download] 100% of  299.51MiB in 00:02:05 at 2.38MiB/s      
[download] Destination: How To Fix Noisy Laptop Fan TOSHIBA L750-1V2 l TOSHIBA PSK2YE Notebook Cleaning Fan Cooler [OlNpTjepfb8].f140.m4a
[download] 100% of   11.71MiB in 00:00:06 at 1.82MiB/s   

Can you tell me why this is necessary? I am worried that this process is not working as it should. Can you confirm that this is what I should expect to hapen?

Regards,

Ken.

[plinej]

@keniv, that is what youtube-dl does. It downloads the audio and video streams separately and then combines them together.

[keniv]

@plinej
OK, thanks for that. Something else I now know I didn't know before.

Regards,

Ken.

[plinej]

Updated to 0.41 - Added ability to edit file name with workupload. Added support for mediafire and google drive. For the best support of google drive install gdown https://github.com/wkentaro/gdown . Added better support for mega.nz with megatools from https://megatools.megous.com/ . Added support for instagram, facebook and twitter video with youtube-dl (or yt-dlp).

[stemsee]

hi

lots of additions! Will sourceforge be possible?

If you try to download this file or a file that doesn't exist it appears to get created.
Try with this
http://ftp.gnu.org/gnu/glibc/glibc-2.35.tar.gz
Is it the http and ftp in the same string, maybe?

[plinej]

@stemsee, That link works for me. It should work for you. Sourceforge seems to download but the file name is messed up. I'll look in to that...

-edit- I think I have sourceforge figured out so 0.4.2 will include that update...

[plinej]

updated to 0.4.2 - Added more options to preferences, including buttons to install or guide you to the web page to install gdown & megatools. Added script to download yt-dlp if you don't have youtube-dl or yt-dlp installed and click ok. Possible support for sourceforge (seems to work in my minimal testing).

[plinej]

updated to 0.4.3 - If your download ends up being a html file and you have yt-dlp or youtube-dl installed the script will retry downloading using that instead. I've noticed it seems to work on alot of video sites. I've also edited the scripts to be a bit less puppy specific so it should work on other distros.

[Jasper]

@plinej

I admit I have not tried your application, but many thanks for sharing as always.

One thing that I have occasional errors myself is with is an issue with SSL certificates, which necessitate having to use

--no-check-certificate

is this something you have addressed?

[plinej]

updated to 0.4.4 - Added --no-check-certificate to wget options per @Jasper's request. Also added the ability to download video from crowdcast. From Chrome you would need to click inspect -> Element -> copy element (the link that shows the m3u8 file. You'll need yt-dlp or youtube-dl for crowdcast downloads to work. I'm sure you could find the m3u8 link with firefox as well but I haven't tried yet.

[Jasper]

Brilliant and I am sure it will be useful

[plinej]

updated to 0.4.5 - Added support for soundcloud with yt-dlp or youtube-dl

[Jasper]

@plinej
Thanks for the inclusion of Soundcloud ........... I think that MixCloud should also work (?)

[plinej]

@Jasper, mixcloud might work but I'll look into it...

--edit--

I see it's subscription based so I'm not sure.

[Jasper]

@plinej

Your are right, you do need to have an account (*free).

BTW for me Aria2c works for downloading. Is your decision to use Wget based on that it is included in all Puppy builds?

The application works well, many thanks

BTW .......... does the script start from "boot up"? or do I need to invoke it to start working?

[plinej]

@Jasper, There are a few reasons for just using wget but being included in just about every distro it seems to be the best option. You have to start it from the menu but you could put it in your /root/Startup folder if you just want it to start running on boot up. Also, looking at the image you posted about downloading from mixcloud using yt-dlp, my script should download that no problem with the default options since it'll attempt to download using wget and if a HTML file is downloaded it will try to download with yt-dlp. I can add mixcloud to use yt-dlp or youtube-dl without trying wget first for the next update.