pm2ubcd script discussion

Discussion/announcements about test/beta releases of UBCD will be posted here.

Moderators: Icecube, StopSpazzing

Message
Author
ady
Posts: 832
Joined: Sat May 08, 2010 5:26 am

Re: pm2ubcd script discussion

#21 Post by ady » Sun May 05, 2013 2:42 am

Just for the record, Explorer and I continued by PM's the POSIX-compatibility issue to avoid cluttering this topic.

Feedback (regarding beta2) is still welcome.

ady
Posts: 832
Joined: Sat May 08, 2010 5:26 am

Re: pm2ubcd script discussion

#22 Post by ady » Mon May 13, 2013 8:29 am

pm2ubcd script beta3 attached to this post.
[UPDATE]
pm2ubcd2013JUN25 attached to post#43
[/UPDATE]

Beta3 supports paths containing space characters.

As before, the "
./ubcd/tools/win32/pm2ubcd/pm2ubcd.txt
" file is relevant for both, Linux and Windows users.

Windows users need all files in the archive, including those under the Linux directory.

Please test and provide feedback.

Code: Select all

FILE : pm2ubcd2013MAY13.tar.gz
SIZE : 7'285 bytes
MD5  : 0c4f61b6165283b10098cb7175ab4c7a
SHA1 : 6e9c0c94938139c168cd926107b21f5844aa5159
[/s]
Last edited by ady on Mon Jun 24, 2013 1:11 pm, edited 1 time in total.

StopSpazzing
Posts: 462
Joined: Tue Sep 09, 2008 4:37 pm
Location: California, USA
Contact:

Re: pm2ubcd script discussion

#23 Post by StopSpazzing » Mon May 13, 2013 1:54 pm

Awesome going To test this out tonight :)
~Just StopSpazzing~

Visit the UBCD Wiki: http://wiki.ultimatebootcd.com
Please check your UBCD ISO MD5 Hash Sum; May prevent issues later on by not having an exact copy.

Currently Working on Common Issues and Repair Tips on the Wiki.

Explorer09
Posts: 178
Joined: Fri Jun 17, 2011 11:23 pm

Re: pm2ubcd script discussion

#24 Post by Explorer09 » Mon May 13, 2013 4:57 pm

Code: Select all

        PMAGIC_ROOT=`echo $1 | sed -e 's|[\]|/|g'`
        ;;
    2)
        PMAGIC_ROOT=`echo $1 | sed -e 's|[\]|/|g'`
        UBCD_ROOT=`echo $2 | sed -e 's|[\]|/|g'`
        ;;
    3)
        PMAGIC_ROOT=`echo $1 | sed -e 's|[\]|/|g'`
        UBCD_ROOT=`echo $2 | sed -e 's|[\]|/|g'`
        ISO_NAME=`echo $3 | sed -e 's|[\]|/|g'`
@Ady

I found unescaped backslashes in the sed syntax here. It should be this

Code: Select all

| sed -e 's|[\\]|/|g
or this:

Code: Select all

| sed -e 's|\\|/|g

ady
Posts: 832
Joined: Sat May 08, 2010 5:26 am

Re: pm2ubcd script discussion

#25 Post by ady » Tue May 14, 2013 12:40 am

Explorer09 wrote:

Code: Select all

        PMAGIC_ROOT=`echo $1 | sed -e 's|[\]|/|g'`
        ;;
    2)
        PMAGIC_ROOT=`echo $1 | sed -e 's|[\]|/|g'`
        UBCD_ROOT=`echo $2 | sed -e 's|[\]|/|g'`
        ;;
    3)
        PMAGIC_ROOT=`echo $1 | sed -e 's|[\]|/|g'`
        UBCD_ROOT=`echo $2 | sed -e 's|[\]|/|g'`
        ISO_NAME=`echo $3 | sed -e 's|[\]|/|g'`
@Ady

I found unescaped backslashes in the sed syntax here. It should be this

Code: Select all

| sed -e 's|[\\]|/|g
or this:

Code: Select all

| sed -e 's|\\|/|g
Before mentioning anything else... For those testing this beta3, don't worry :). It is safe to keep testing. To be able to trigger any difference in the result, the path(s) would need to contain some very specific special characters, and even then, the *resulting* path is verified (whether it really exists), and all the resulting paths are displayed for explicit user's confirmation, before executing any action at all.

______

FWIW...

Between beta2 and beta3, I changed the sh script several times. This specific code indeed needs the "double" (aka. "escaped") backslash character. In practice, after every change I made to these lines of the script (and I changed the logic of these lines of the code *many* times for many different trials), the result was "the same", independently of using one or two backslash characters (which I modified many times according to the specific sed code I was using): either the obtained path was correct, or it wasn't.

Finally, I partially "moved" the code to handle "backslash characters in paths" into the cmd script. I left this sed code in the sh script, just in case (for some reason) some backslash character gets to that point in the code, which could potentially change the resulting paths. I almost forgot about it :oops: .

Now, I should either:
A_ correct this sed code (with escaped backslash, and with the square brackets for easier reading); or,
B_ delete this sed code, leaving only the definition of variables.

There are enough steps where the user (and/or the script) can (be forced to) avoid using the wrong paths. The intention is to have an additional ("extra") check, and if the user is not trying to screw the process, this sed code is not even necessary. In other words, it is there so to make the script a little bit more "foolproof".

Whichever the original intention, I can see a potential case where this incorrect syntax could make a difference (and even in such very rare case, the user still would need to explicitly accept the values before anything actually could happen).

I'll correct this sed syntax for the next release. Thanks.

Explorer09
Posts: 178
Joined: Fri Jun 17, 2011 11:23 pm

Re: pm2ubcd script discussion

#26 Post by Explorer09 » Tue May 14, 2013 8:39 pm

ady wrote:Finally, I partially "moved" the code to handle "backslash characters in paths" into the cmd script. I left this sed code in the sh script, just in case (for some reason) some backslash character gets to that point in the code, which could potentially change the resulting paths. I almost forgot about it .

Now, I should either:
A_ correct this sed code (with escaped backslash, and with the square brackets for easier reading); or,
B_ delete this sed code, leaving only the definition of variables.

There are enough steps where the user (and/or the script) can (be forced to) avoid using the wrong paths. The intention is to have an additional ("extra") check, and if the user is not trying to screw the process, this sed code is not even necessary. In other words, it is there so to make the script a little bit more "foolproof".
Ady, would you mind that I post my argument here?
IMO, the sed code does not make the script any more "foolproof" at all. And I would suggest removing it from the sh script. Here are the reasons:
1. The "backslash-as-path-separator" confusion only happens in Windows. Linux/Unix users aren't supposed to make such mistake.
2. If they did make a mistake, then the shell will interpret the backslash first, before your sed code processes. This might be hard to understand, so look at the example below.
Suppose on Linux the user runs the script like this in the terminal

Code: Select all

explorer$ ./pm2ubcd.sh \home\explorer\new-folder\test\pmagic
The shell will interpret and execute this: (note the newline and tab)

Code: Select all

./pm2ubcd.sh \home\explorer
ew-folder[TAB]est\pmagic
And a "command not found" should appear. The Windows port of Bash (Cygwin, MSYS, etc.) might be tolerant to this syntax, but it will be inappropriate to do the slash conversion in the sh script.

This will work with your sed code:

Code: Select all

explorer$ ./pm2ubcd.sh \\home\\explorer\\new-folder\\test\\pmagic
However I think the person who types the path this way will already know what he is doing.
Last edited by Explorer09 on Wed May 15, 2013 8:19 am, edited 1 time in total.

ady
Posts: 832
Joined: Sat May 08, 2010 5:26 am

Re: pm2ubcd script discussion

#27 Post by ady » Wed May 15, 2013 3:22 am

Explorer09 wrote:1. The "backslash-as-path-separator" confusion only happens in Windows.
Correct. But there is no method to prevent someone from modifying the cmd script as if it were independent of the sh script (and they are not 100% independent). And that's the potential (future) problem. It would be enough with very simple changes to the cmd script, and the backslash could be back. Granted, there are several simultaneous conditions to be met so to trigger the potential problem, and even then, nothing can happen unless the user explicitly accepts the values.

Had I started directly with the cmd script handling these cases, probably I wouldn't had added these sed lines. But once it is there and since I have already thought about the potential future situation, then I would tend to think that there is no reason to erase that code. In the current state, it is not changing the resulting path, and the "extra" running time is not noticeable.

I have already added several precautions against a user typing in "crap" arguments to the command line. This is even more than what I was initially planning to do, since I think of this script as a simple help, not as replacement for laziness or for users not knowing what they are doing. In most cases, typing in the first argument alone is enough, and the accompanying text file has enough recommendations and tips. As I mentioned before, anything that happens, or doesn't happen, is all up to the user's responsibility, exclusively.

So, when I thought about "foolproof" regarding these sed code, I was thinking more about potential future changes to the code (possibly by someone else), and not about the final user. It is just another barrier so to reduce the chances of passing the wrong paths, and it doesn't seem to have any noticeable negative effects.

Explorer09
Posts: 178
Joined: Fri Jun 17, 2011 11:23 pm

Re: pm2ubcd script discussion

#28 Post by Explorer09 » Wed May 15, 2013 8:48 am

ady wrote: [T]here is no method to prevent someone from modifying the cmd script as if it were independent of the sh script (and they are not 100% independent). And that's the potential (future) problem. It would be enough with very simple changes to the cmd script, and the backslash could be back.
[......]
So, when I thought about "foolproof" regarding these sed code, I was thinking more about potential future changes to the code (possibly by someone else), and not about the final user. It is just another barrier so to reduce the chances of passing the wrong paths, and it doesn't seem to have any noticeable negative effects.
My idea is to remove redundancy in the code, and to be honest to the future code editors. If a part of code is not always working (in a given condition), I think it will be better to let it always fail, rather then pretending it works. If someone changes a little thing in the cmd script, and that change bring the backslash back, then that will be not our fault. It would be he that breaks such dependency.

If I were you, I would just document in the sh script the potential problem when handling backslashes, and I won't try to add a fix that is never going to be complete (i.e. the sed way).

ady
Posts: 832
Joined: Sat May 08, 2010 5:26 am

Re: pm2ubcd script discussion

#29 Post by ady » Wed May 15, 2013 11:08 pm

I understand the intention of avoiding unnecessary code, but I'm still not convinced that this particular
sed
code is really excessive when we consider the context.

For example, this same
sed
code would be useful in case someone would want to expand the script with additional PMagic's boot parameters. The first two command line arguments of the script must be existent paths (and it would certainly help if Windows users would remember to use the already-accepted slash "/" instead of backslash "\" for paths, but that's only wishful thinking). But then the third command line argument ("iso_filename") can use a non-existent (optional) path. This
sed
code assures that this argument is transferred to the syslinux.cfg file using slash (when the optional path is used for the "iso_filename" boot parameter).

Now, if a new boot code is added to the script (like for example, "directory="), the same
sed
code in question is useful. Instead of having to analyze whether this
sed
code is necessary (or at least potentially helpful) for each boot code (or for each command line argument for this pm2ubcd script), this same structure is simply repeated, only changing the variable name. By consistently using this
sed
code, the extension of the script (to include additional parameters) is easier, specially for non-programers.

For someone that knows about cmd and sh scripts, and also knows about Syslinux, and also knows about PMagic boot parameters, most of this script could be considered unnecessary bloat. Since we don't know if all this simultaneous knowledge is going to be present (and timely available) for UBCD in the future, I am inclined to leave this
sed
code in place.

StopSpazzing
Posts: 462
Joined: Tue Sep 09, 2008 4:37 pm
Location: California, USA
Contact:

Re: pm2ubcd script discussion

#30 Post by StopSpazzing » Fri May 17, 2013 1:42 pm

Test it, works good. So GJ.
~Just StopSpazzing~

Visit the UBCD Wiki: http://wiki.ultimatebootcd.com
Please check your UBCD ISO MD5 Hash Sum; May prevent issues later on by not having an exact copy.

Currently Working on Common Issues and Repair Tips on the Wiki.

ady
Posts: 832
Joined: Sat May 08, 2010 5:26 am

Re: pm2ubcd script discussion

#31 Post by ady » Wed May 22, 2013 8:43 am

I'd like to request some specific feedback.

Beta3 of pm2ubcd changes the way the "iso_filename" boot parameter is inserted into syslinux.cfg. For example, instead of

Code: Select all

iso_filename=ubcd523.iso
the parameter is now inserted as

Code: Select all

iso_filename='ubcd523.iso'
This boot parameter is required to boot PMagic directly as ISO image, and boot it (ie. the ISO image) by using, for example, GRUB2 (or by using any other method of "ISO-booting" that requires the "iso_filename" boot parameter).

It should be noted that the "iso_filename" parameter needs to correspond to parameters used when building the updated ISO image.

Items to be tested:
_ Whether the ISO image successfully boots (first as ISO image and then into PMagic itself until PMagic is fully working) when the "iso_filename" parameter is used, either with or without the two single quotation marks.
_ Whether the default value of the "vmalloc" boot parameter is adequate to boot the "merged" (UBCD + updated PMagic) ISO image (I'd tend to think it would need to be around "vmalloc=544" or perhaps even higher, instead of the current default, but this is part of the tests).

For the record, "ISO-booting" requires, generally speaking, more RAM than when using other methods.

If someone could test and provide feedback about using pm2ubcd, building a new ISO image and then "ISO booting", it would be very helpful.

TIA,
Ady.

StopSpazzing
Posts: 462
Joined: Tue Sep 09, 2008 4:37 pm
Location: California, USA
Contact:

Re: pm2ubcd script discussion

#32 Post by StopSpazzing » Sun May 26, 2013 10:23 pm

Should take me about >1 minute to do on my main computer (16gb ram, quadcore 4ghz). So to understand you correctly, use pm2ubcd to merge latest PMagic,then create the iso afterwards using ubcd2iso, and test booting from it, correct?
~Just StopSpazzing~

Visit the UBCD Wiki: http://wiki.ultimatebootcd.com
Please check your UBCD ISO MD5 Hash Sum; May prevent issues later on by not having an exact copy.

Currently Working on Common Issues and Repair Tips on the Wiki.

ady
Posts: 832
Joined: Sat May 08, 2010 5:26 am

Re: pm2ubcd script discussion

#33 Post by ady » Mon May 27, 2013 1:54 am

StopSpazzing wrote:Should take me about >1 minute to do on my main computer (16gb ram, quadcore 4ghz). So to understand you correctly, use pm2ubcd to merge latest PMagic,then create the iso afterwards using ubcd2iso, and test booting from it, correct?
Well, it depends on what exactly you are calling "booting from it".

1_ The "iso_filename" parameter needs to match the real name (to be used in ubcd2iso too) and location of the ISO image in the hard disk (which is also used in the GRUB2 entry for "ISO-booting").
2_ The "vmalloc" value might also influence on whether "ISO-booting" PMagic is fully successful.
3_ To be able to conclude that "ISO-booting" PMagic is successful, seeing PMagic's boot menu from inside the ISO image located in the hard disk is _not_ enough. Once the boot menu inside the ISO image is displayed, PMagic needs to be actually booted. Only if PMagic itself is actually successfully 100% working, then the test result is OK.
4_ Repeat the "ISO-booting" test using the original (official) UBCD ISO image, "as-is", with the adequate GRUB2 entry, and then compare results.

The effect of "iso_filename" and "vmalloc" are only tested when "ISO-booting" (up to actually seeing PMagic's GUI desktop). When booting PMagic with the structure already expanded (on optical media or on UFD or on HDD), the 2 parameters in question are ignored.

What I want to find out is whether the resulting "iso_filename" and "vmalloc" parameters are adequate.

TIA,
Ady.

StopSpazzing
Posts: 462
Joined: Tue Sep 09, 2008 4:37 pm
Location: California, USA
Contact:

Re: pm2ubcd script discussion

#34 Post by StopSpazzing » Mon May 27, 2013 1:32 pm

IF "%~1" NEQ "" GOTO _ARGS_1
bash --norc pm2ubcd.sh
GOTO _EOF


Bash doesn't work in windows...so it give me an error.
~Just StopSpazzing~

Visit the UBCD Wiki: http://wiki.ultimatebootcd.com
Please check your UBCD ISO MD5 Hash Sum; May prevent issues later on by not having an exact copy.

Currently Working on Common Issues and Repair Tips on the Wiki.

ady
Posts: 832
Joined: Sat May 08, 2010 5:26 am

Re: pm2ubcd script discussion

#35 Post by ady » Mon May 27, 2013 11:09 pm

StopSpazzing wrote:Bash doesn't work in windows...so it give me an error.
@ StopSpazzing,
I thought you had already executed the script before, without problems. So what's different this time? Could you please provide more details?

***
@ All,

I want to remind Windows users...

_ Both "Linux" and "Win32" branches contain scripts, and _all_ files are needed, including those under the Linux directory structure.

_ Some Windows-based antivirus and security-related tools might "silently" block bash.exe (among others), triggering strange errors seen in the Windows command prompt. Unless this specific file (bash.exe) was somehow infected in such specific system, this situation is a false positive. In some cases, a "recent" update of the antivirus database might be now triggering this blocking action (while this behavior might had been different in the past).

StopSpazzing
Posts: 462
Joined: Tue Sep 09, 2008 4:37 pm
Location: California, USA
Contact:

Re: pm2ubcd script discussion

#36 Post by StopSpazzing » Sun Jun 09, 2013 10:29 am

Yea, I fixed the problem, was in the wrong folder.


It boots just fine after merge, cleanup, then running the ubcd2iso to compile the iso. Only thing I noticed was that the Pmagic Version number menu item in UBCD didn't match the Parted Magic version that was detected with your script. Would be awesome if that was fixed ;)

Curious, why not just have one script connect to the other script? So after merge, have option to choose one of the other commandline/shell scripts to, for instance, run ubcd2iso with the parameters that were added?
~Just StopSpazzing~

Visit the UBCD Wiki: http://wiki.ultimatebootcd.com
Please check your UBCD ISO MD5 Hash Sum; May prevent issues later on by not having an exact copy.

Currently Working on Common Issues and Repair Tips on the Wiki.

Explorer09
Posts: 178
Joined: Fri Jun 17, 2011 11:23 pm

Re: pm2ubcd script discussion

#37 Post by Explorer09 » Sun Jun 09, 2013 11:23 pm

StopSpazzing wrote: Curious, why not just have one script connect to the other script? So after merge, have option to choose one of the other commandline/shell scripts to, for instance, run ubcd2iso with the parameters that were added?
This will increase coupling between two scripts, and decrease modularity.

I disagree with your idea.

ady
Posts: 832
Joined: Sat May 08, 2010 5:26 am

Re: pm2ubcd script discussion

#38 Post by ady » Tue Jun 11, 2013 10:23 am

StopSpazzing wrote:Yea, I fixed the problem, was in the wrong folder.
The adequate folder is essential, as noted in the txt file (relevant for both, Linux and Windows). The pm2ubcd script uses similar concepts or conditions that are already in use in other scripts included in UBCD (not necessarily with the same type of code).

I'm glad you solved that issue.
It boots just fine after merge, cleanup, then running the ubcd2iso to compile the iso. Only thing I noticed was that the Pmagic Version number menu item in UBCD didn't match the Parted Magic version that was detected with your script. Would be awesome if that was fixed ;)
Also mentioned in the txt file :wink: .

The version of PMagic included in the official releases of PMagic is included by Victor (manually, I would guess). If a user manually updates any tool in UBCD, the version needs to be manually corrected too, and this is also valid for PMagic.

In the particular case of PMagic in UBCD, the version number that is displayed in the initial boot menu comes from "main.cfg" (for the default Syslinux menu). As mentioned in the pm2ubcd.txt file, only the relevant files inside "/pmagic/" are touched, which leaves out "main.cfg".

There is a reason for this limitation: customizations. If the script would edit main.cfg with the new version of PMagic, the only way to be sure about the result would be to apply the script to the original official UBCD. So for each time you would want to use pm2ubcd, you would need to start over, from scratch.

If a user has additional customizations, specially if they are part of "main.cfg", how the script would correctly identify what exactly to change and what exactly *not* to modify? The reason pm2ubcd can do what it does is because the basic structure and the expected patterns of both, PMagic and UBCD, are known. The script can handle many variations of PMagic, and it won't touch any personalized file nor any PMagic's module (such as those included in UBCD, for example). But, for that logic to work as expected (and remember, we have no control about future versions of PMagic, and yet, the script will mostly keep handling it OK), we avoid touching anything else that the user might (or might not) have customized.

Giving that the correct version number is included in the boot menu of PMagic itself (just select PMagic from UBCD and the default menu entry shows the version number in the lower part of the screen), my suggestion would be to avoid specifying any version number of PMagic in "main.cfg". Of course, this is up to Victor.

The current possibility, for the final user that executes pm2ubcd, is to either "leave it as is" (since PMagic will work correctly anyway and the actual version is mentioned in its own menu), or to edit main.cfg manually (which is exactly the same as with any other customized tool or manual update in UBCD).
Curious, why not just have one script connect to the other script? So after merge, have option to choose one of the other commandline/shell scripts to, for instance, run ubcd2iso with the parameters that were added?
The goal of the pm2ubcd script is just to make it easier for users, since PMagic is generally updated more frequently than UBCD (even Victor could use it as part of the building procedures of future releases of UBCD, if he wants). It is not meant to be an "all-in-one" solution (for example, it doesn't include a "directory=" boot parameter), and it is certainly not a replacement for "laziness" :mrgreen: (to be absolutely clear, I'm not implying anything about anyone; I'm just stating that those two situations are not included in the objectives of pm2ubcd).

It is not so simple to always find an adequate balance between needed features, usefulness, ease of use, and flexibility.

For customizations, I would suggest using the scripts (probably to be executed in this order): pm2ubcd, antivirus updates, syslinux2grub4dos, ubcd2iso, ubcd2usb; each with the adequate parameters (sometimes the same value needs to be used in more than one script).

Explorer09
Posts: 178
Joined: Fri Jun 17, 2011 11:23 pm

Re: pm2ubcd script discussion

#39 Post by Explorer09 » Tue Jun 11, 2013 9:51 pm

Regarding the PMagic version number in "main.cfg", I think it is good to display a warning that "you have to change it manually" in the script.

When I helped ady making the pm2ubcd script, I added a variable (PMAGIC_VERSION) that displays the PMagic version, just for this purpose.

So, here is an example message. Put it after where the script is "Done".

Code: Select all

echo "Please note that the PMagic version number in the isolinux boot entry is"
echo "NOT UPDATED automatically. To update it,"
echo "1. Open ubcd/menus/syslinux/main.cfg in a text editor"
echo "2. Search for the string \"Parted Magic\" and replace the version number"
echo "   after it with \"$PMAGIC_VERSION\" ."

ady
Posts: 832
Joined: Sat May 08, 2010 5:26 am

Re: pm2ubcd script discussion

#40 Post by ady » Tue Jun 11, 2013 11:33 pm

As with any other program that is manually updated or customized (as opposed to being part of the official releases of UBCD), the user has to first think and then decide what to do. Why we would want to make it differently in the case of PMagic in UBCD?

The info in the pm2ubcd.txt file already mentions this matter. I could understand someone suggesting "the txt file should mention the need to manually update the version number of PMagic in main.cfg in a more explicit way" (because the text is more generic than that). I would probably have a different point of view, but I would still understand this different opinion regarding the content of the txt file.

But the script is supposed to be relevant to the functionality. It only simplifies the task of updating PMagic in UBCD, for someone that knows what he is doing. It is still a customization, just as any other manual update. How many details we should add to the (already big) script that are not part of the main functionality? Isn't enough to add this type of information in the txt file?

Instead of adding information to the script - information which, in this case, is not going to change the functionality of the program nor improve the usage of the script itself - why not add "It is recommended to read the pm2ubcd.txt file"?

Furthermore, if we fear that someone would (wrongly) think "Hmm, I'm confused, which version was updated? Was it really updated? It seems it failed to update?", then the specific version of PMagic should never be mentioned in "main.cfg" (nor "main.lst"). IMO, this is the more adequate solution for PMagic (other programs in UBCD are not updated as frequently). Remember: the version number is correctly displayed before booting PMagic, in the first entry of its own boot menu. No need to manually edit main.cfg, no need to read or clarify, and it would still be accurate.

If someone is so lazy to even read a text file, I would still want to help him with UBCD or with a technical utility such as PMagic. And the best help I can offer to such individual would be "don't even touch it, give your hardware to a technician".

Clarity is very important. If pm2ubcd.txt is not clear for the target user, let's improve it.

Then if someone would still be confused with the version of PMagic in main.cfg, I would say "read the txt file and you are free to customize main.cfg as you wish". I don't mean it in a rude way; just that the user has his hands free, the same as with any other customization.

If someone has a problem with ISO-booting for example, I would encourage such user(s) to provide feedback here so we can gladly improve the script. That's about functionality.

Post Reply