Is it possible to host the latest EXPLODED ISO of UBCD?

[milindck]

I have downloaded and installed UBCD 5.3.0 onto a USB.

Now here is my issue.
Every time the new UBCD minor version is published I would like to only download a few updated files

Since "wget -N <BASE_URL>" is capable of downloading only the newer files i would like to avoid downloading the big xxxMB ISO file.

Over time some files may be deleted from UBCD and to accomodate that maybe a script (script_to_delete_obsolete_files.sh) could be written to delete those files.

Using command line something like

Code: Select all

mount -t vfat /dev/sdb1 /media/sdb1
wget -N -P /media/sdb1 http://www.serverninjas.com/sn/ubcd_latest
/media/sdb1/<script_to_delete_obsolete_files.sh>

And then for those who need the ISO file they can run the following script to build one using the ubcd2iso.sh script

Code: Select all

/media/sdb1/ubcd/tools/linux/ubcd2iso/ubcd2iso.sh /media/sdb1 /tmp/UBCD.iso

In my opinion this will significantly reduce the bandwidth required to update the UBCD contents for everyone.

[Victor Chew]

The idea is a sound one.

I hope someone with the server capacity can step it to do this.

Actually rsync might be a better match, since it can perform deletion of obsolete files at the same time.

[milindck]

...
Actually rsync might be a better match, since it can perform deletion of obsolete files at the same time.
...

Perfect, me not being a unix guy do not know that. But I just tried to read the man pages and it looked like the rsync needs rsync daemon/server

Plus I am more tempted to request hosting Exlploded UBCD ISO files as a sourceforge project (i do not know if this is even practical as per sourceforge rules as it will involve hosting binaries for the most part instead of sources) which can then be downloaded using GIT and even allow going back in time to download deleted but desired binaries that were included in past versions.

[ady]

Generally speaking (and FWIW), I am not "comfortable" with distributing scripts to delete / replace files. The risk of deleting some (part of a) customized version is always there.

@Victor,
For future versions, perhaps you might want to consider providing xdelta downloads in addition to whole ISO images, as you used to. It is not "helpful for the lazy ones", but it might help with bandwidth.

[The Piney]

Generally speaking (and FWIW), I am not "comfortable" with distributing scripts to delete / replace files. The risk of deleting some (part of a) customized version is always there.

@Victor,
For future versions, perhaps you might want to consider providing xdelta downloads in addition to whole ISO images, as you used to. It is not "helpful for the lazy ones", but it might help with bandwidth.

If any new scripts that do delete files are added, a disclaimer or Warning! could be added to the download locations or even put into the script before starting any process. I can understand why you may not be comfortable with it.

If you create the xdelta files and upload those a day or so ahead of the ISO release, those of us that seed the torrents could carry a bit more of the load. On the previous subject (deleting files), xdelta deletes and replaces files doesn't it? A vanilla UBCD ISO needs to be used with the xdelta file for the hash to come out right AFAIK. Wouldn't a customized version get messed up?

[milindck]

Generally speaking (and FWIW), I am not "comfortable" with distributing scripts to delete / replace files. The risk of deleting some (part of a) customized version is always there.

Hey ady,

Are you familiar with GIT?

[ady]

If any new scripts that do delete files are added, a disclaimer or Warning! could be added to the download locations or even put into the script before starting any process. I can understand why you may not be comfortable with it.

The problem is less relevant for users that actually *read* (although, even after reading, mistakes happen ). But newbies and "lazy" users (and there are too-many out there ) tend to "trust" that everything is going to work as "they" expect, with no mistakes, no errors, no misunderstandings, and that "their files" are %100 safe, no matter what, no back up needed.

I wouldn't be surprised if a user were to use a script (or two) to download some files and then patch their previous UBCD with those downloaded files, just to sign up in the forum to complain and ask how to recover his files or prior functionality.

If you create the xdelta files and upload those a day or so ahead of the ISO release, those of us that seed the torrents could carry a bit more of the load.

I'm not against the idea, but in practical terms there seems to be no need. In some occasions, Victor has even released the ISO images to the http/ftp mirrors before posting the torrent, and AFAIK it was all OK anyway. Also popular software repositories have UBCD almost immediately available just a few hours after release.

The issue, IMHO, is more related to users downloading hundreds of MiB when there are just a few MiB of relevant difference.

The xdelta method is not ideal for UBCD either; I just mentioned it because Victor used to use it in the past in certain circumstances.

On the previous subject (deleting files), xdelta deletes and replaces files doesn't it? A vanilla UBCD ISO needs to be used with the xdelta file for the hash to come out right AFAIK. Wouldn't a customized version get messed up?

Indeed. xdelta would be worth only if all the following conditions are met:

• The user has the prior version of UBCD as originally distributed; and,
• The user downloads and applies the xdelta diff to the prior version so to obtain the new one; and,
• The xdelta diff is relatively small.

The last point is relevant because providing a diff (whether xdelta or in any other format / method) also requires resources and it is not as easy as downloading the whole ISO image and moving on. So the bandwidth users would save (by not having to download the whole ISO image) has to be meaningful.

Although the risk of messing up with the xdelta method indeed exists, the newbies and lazy ones would tend to avoid it. They would tend to just download the whole ISO. Which brings up the simpler (and already-available) solution... torrents.

Similarly to xdelta, a user can copy the prior ISO image file (as originally released, and even if not so), and point the new torrent to this copy. The new torrent will accept the pieces that are an exact match, and download the ones that are not.

This torrent method would be much more efficient if the files (of the UBCD ISO image) would be already expanded, i.e. one torrent including each individual file and directory structure of UBCD, instead of one torrent of one ISO image.

In past discussions, Victor has been clear about his preference towards distributing one (and only one) whole ISO image. In the past, I have suggested the possibility of not including the antivirus databases, thus leaving the task for users (by means of the included scripts, among other available methods).

So, IMHO, a torrent with the UBCD directory structure and all files would be the easiest solution. Users wanting to build a new ISO image based on the new set of files, could use the ubcd2iso scripts.

The caveat of the torrent method is that the user "needs", either a copy of the UBCD structure (or the ISO image), or to "disconnect" the torrent from the UBCD structure. This is not a technical requirement, but more of a practical one. Imagine having the UBCD structure and working from it, without a copy. Then, intentionally or not, the torrent is started. Whichever change the user might had performed would be overwritten by the torrent trying to download new pieces that are no longer a match. OTOH, this is a caveat of xdelta (and of almost any other method) too, in some form of another.

Since probably Victor would like to keep providing the whole ISO image (from the http/ftp mirrors and with a p2p torrent), the next UBCD release announcement could include a second torrent with the individual files and directory structure, perhaps in the forum only (not in the main download page).

BTW, the UBCD wiki already includes an example of how this torrent method would be used. See (the last section of) http://wiki.ultimatebootcd.com/index.ph ... g_Hash_Sum

Are you familiar with GIT?

I think git would be too much. Victor would need to have a public git repo (and maintain it), and users would need to know how to use it. See my previous paragraphs for an alternative possibility.

[Victor Chew]

@milindck: git is not an option. See this:

In addition, we place a strict limit of files exceeding 100 MB in size.

@ady: xdelta does not directly address the issue of updating the USB memory stick directly. As a side note, I stopped generating the xdelta file for two reasons. 1) There are very few downloads. 2) For some releases (eg. when we were updating Parted Magic), the xdelta file too large to be worth it. So the xdelta solution is just too complicated except to a few people. You need to have the right ISO image, the right software installed, the right skill to apply the xdelta file. Bottom-line, 99.9% of the people don't care!

@ady: Again, don't see how a torrent of an expanded UBCD ISO will help directly with USB memory stick update.

I still think rsync is the best solution. It directly addresses the USB memory stick issue. It is available on both Linux and Windows. It is very easy to setup an anonymous, read-only rsync server. The only issue is bandwidth capacity.

Heck, I will set one up later and post details. I doubt many people will use it, but I will monitor the bandwidth usage closely.

[Victor Chew]

Done. Change current directory to directory on USB memory stick where UBCD resides, then issue this command:

Code: Select all

rsync -rP www.ultimatebootcd.com::ubcd/* .

[ady]

@ady: xdelta does not directly address the issue of updating the USB memory stick directly.

As I said, I only mentioned it because it was used in past versions. I also said it was not a good solution in this case.

@ady: Again, don't see how a torrent of an expanded UBCD ISO will help directly with USB memory stick update.

As explained in the UBCD wiki and in my post above.

Generally speaking, directly updating on the USB stick is not recommended anyway. It is better to have the directory structure on fixed HDD. Once the update is complete, then copy to USB.

I'll give an example. Say only

ubcd/images/fdubcd.img.gz

was updated and a new torrent is released. If the torrent includes the whole ISO image, users have to download it as a whole (hundreds of MiB).

If, instead, the directory structure is included, the user "points" the torrent to (a copy of) his own UBCD directory structure, and the actual download will be only the unmatching pieces, i.e. fdubcd.img.gz (and 2 more torrent pieces).

This is the equivalent of selecting only part of the files in a torrent and downloading only those. As a general rule, this would be considered counterproductive for the ("life of the") torrent (since only parts of it are shared), but since the rest of the files are already downloaded (from prior UBCD versions), this shouldn't be a problem. That is, unless torrent users would actively unselect the other files so to block them for upload.

@Victor, you could try this in the next release, as an alternative torrent (posted on the forum only, not in the download page). Nothing prevents you from doing rsync and this alternative torrent too.