VERSION 0.1 ALPHA Release date 02/April/2005
by Jacopo Lazzari
EVERYTHING YOU EVER WANTED TO KNOW ABOUT BOOTING FROM AN USB PENDRIVE … AND DARED TO ASK
Scope of this article is to publish a summary of my experiences in the search for a reliable, accurate and practical way to manage USB Pendrives (a.k.a. USB Stick, USB Thumbdrive, USB Memory, USB Keyphob, USB Keyfob), to FORMAT them, and to make them BOOTABLE.
The above has been done using ONLY Public Domain, Freeware, Open Source or however freely available software.
I am perfectly aware that there is COMMERCIAL software that “might” do the same things depicted here, possibly in a simpler way, but my personal belief is that every effort should be made to try and use what some people has resolved to share freely, BEFORE spending money to buy something you’ll probably use just a few times.
Following you will find some description of the findings and of the proved ways to accomplish this goal, though I have tried to make the article as simple as possible, the intended audience is people that ALREADY have a more than basic knowledge on the following topics:
- Standard IPL (Initial Program Loader) and/or MBR (Master Boot Record)
- BIOS (Basic Input Output System)
- Hard Disk geometry, formatting and partitioning
- Windows 9x/ME and NT/2K/XP/2003 filesystems and related issues
Anything hereafter derives from my personal experience, from what I learned from various sources on the Internet (original source quoted whenever possible) and is, to my best knowledge, accurate and working, at least on MY hardware.
Putting into practice what I relate here CAN destroy data on your Hard Disk, render your system unbootable, and it definitely WILL do so if you make just a tiny small error in using the utilities I suggest.
SO use of this article is RESERVED to people that:
- have a GOOD knowledge of Windows based PC’s
- BACKUP their data before doing this kind of modifications
- do not fear the possible time losing an error could lead to
- do not wish to complain with me if anything goes wrong
Though every reasonable effort has been made in testing this script, the Author accepts no liability whatsoever. This info may or may not help you do whatever it is supposed to do, it could even possibly completely destroy your software, hardware and set your house/office to fire.
No warranty implied, not even that of fitness for any particular purpose apart taking up a little bit of your hard disk space, should you want to save it locally.
To put it in plain English, if you just skipped the above section, three rules apply:
- YOU ARE ON YOUR OWN
- YOUR MILEAGE MAY VARY
- DON’T COMPLAIN WITH ME
Information hereby provided is released as CAREWARE, meaning that whenever someone uses it, misuses it, learns something from it, derives work from it, or whatever, he/she performs ANY of the following
1 – Smiles at somebody/something
2 – Hugs or pats on the shoulder a friend/colleague/pet/furry little creature from Alpha Centauri
3 – Acts nicely towards a friend/foreigner/alien/any other living being
4 – Kisses his/her girlfriend/wife/boyfriend/husband/any other living being (de gustibus…)
5 – Feels happy even without apparent reason
6 – Stops whining for an hour, a day, a week, your choice
Important Note: if you don’t like this idea, just ignore it — you can have this anyway.
That’s one way to distinguish the world of ideas from the rest of human history:
you can disregard an idea …
…and no one knocks on your door at midnight.
(Unless it’s your neighbour in need of a cup of sugar), in which case you MAY act NON-nicely as defined below:
There are ongoing threads on specialized forums on the Internet, among which I would like to point out the following two:
about how it is possible to make an USB Pendrive bootable, and how to install to it various Operating Systems of the Microsoft Windows family.
I am just a very basic Linux user, so the argument will NOT be treated here, hoping that someone with sufficient knowledge has already, or soon will have, written a similar HOWTO for the Linux family of Operating Systems.
Moreover, I am not going to relate anything about what actually happens once the MBR and the BOOTSECTOR have been loaded, and how to avoid the possible errors that can happen after.
The discussion is still open on this topic and as I simply DO NOT own the necessary hardware (i.e. 512 Kb or 1 Mb USB Pendrive) and software,(i.e. licenses for all the Operating Systems) I am not going to experiment further in the near future.
Of course any DOCUMENTED contribution will be gladly accepted and linked to in subsequent releases of this document.
An USB Pendrive is made by three main components:
- An USB connector
- A memory chip, which size varies from 16 Kb, 32 Kb, 64 Kb, (elder models) to 128 Kb, 256 Kb, 512 Kb (common models), up to 1 Gb, 2Gb and even 4 GB (currently quite expensive)
- An USB controller chip
There are several different makes of the controller chip, I haven’t found an utility to identify it.
Here is a page from a manufacturer that gives some information about this:
As you can see from this related page, there are different utilities for different makes of the controller:
A known manufacturer is Prolific:
Another one is COSTAR/OTI:
It seems like the controller chip manufacturers, following simply CCM, give to the actual brand (manufacturer or reseller) of USB Pendrives, following simply BMR, some basic utilities, which the BMR “customizes” in a way that it works ONLY with their certain brand of Pendrives.
If you search enough on the Internet, you will probably be able to find some of these utilities as an unrestricted or however less resticted version, but an utility for a certain CCM will not work on another CCM’s chip.
These utilities generally allow for a “lower level” access to the Pendrive, making it possible to activate some features, like:
- reserving a hidden password protected storage
- making the Pendrive partitioned in two parts, a public one and a password protected one
- making the controller map the Pendrive as two separate drives + a third password protected one
Usage of these features is outside the scope of this paper.
A fourth, fundamental function of these utilities is to write an MBR to the Pendrive and FORMAT it, eventually making it BOOTABLE.
Booting from an USB Pendrive is possible if BOTH the following requirements are met:
- the motherboard of the PC you want to boot it from SUPPORTS USB booting from BIOS (a.k.a. BIOS USB LEGACY support)
- the Pendrive is CORRECTLY formatted
Nothing (except perhaps a BIOS upgrade from the motherboard manufacturer) can be made to correct the first problem.
Even if the motherboard supports USB booting, its implementation may be defective, results and direct observation lead to find that there are motherboards with one or more of the following “defects”:
- motherboards that do not boot from USB when you switch the PC on, but do so after a warm boot (CTRL+ALT+DEL)
- motherboards that do not boot from USB after a warm boot (CTRL+ALT+DEL), but do so when you switch the PC off then on (cold boot)
- motherboards that sometime boot and sometime do not
- motherboards that ONLY boot from USB media formatted as “Superfloppy” (a drive formatted as to have no MBR, but directly the BOOTSECTOR as Sector 0)
- motherboards that ONLY boot from USB media formatted as “Hard Disk” (a drive that has a “normal” MBR and one or more partitions)
- motherboards that only boot from a FAT 16 Primary partition (hex 06)
- motherboards that only boot from a FAT 16 Primary partition (hex 06) or from a VFAT16 (or FAT 16 LBA mapped) Primary partition (hex 0e)
- motherboards that can boot win 9x files but do not with NT/2K/XP/Server 2003 ones (that is why I made the preceding note in the preface about the limit of this paper)
Please let me know if you find some other “strange” behaviour.
About the second problem, there are a few freely available tools that supposedly work, the most used of which is the so-called HP USB UTILITY.
There are two versions of this utility for Windows:
(search for the relevant spxxxxx with Google or on HP site)
sp27213 HP Windows Format Utility for USB Drive Key or DiskOnKey
version 2.00.006 A (6 Feb 04)
sp27608 Windows-based Format Utility for HP Drive Key or DiskOnKey USB Device
version 2.1.8 A (17 May 04)
A DOS one:
sp27214 DOS-based Format Utility for HP Drive Key or DiskOnKey USB Device
version 1.00.012 (12 Jul 04)
And this one with Syslinux on it:
cp004934 HP Drive Key Boot Utility
version 7.11.3790.0 (14 Dec 04)
Dell has a utility to make their keys bootable. R69131.EXE
it looks like the HP one, but it won’t recognize the Pendrives I tested.
Another Dell utility can be found here:
the description text is wrong as it does not have MS-DOS fles on it, but rather FREE DOS ones, we will see this utility later.
And there are a few more floating around, you can maybe find them, or maybe not.
However, NONE of the above (and a few more I tried) were able to CORRECTLY format my USB Pendrives.
There are reports that using the HP version 2.00.006 A utility gives better results than the version 2.1.8 A, but I could not find any difference on my USB Pendrives.
Following I will take as an example my 128 Mb KingMax, USB 2.0 compliant, but I made some tests with borrowed ones, with varying results, which led me to write this paper.
LET’S GET TECHNICAL
The BIOS geometry problem.
From a technical point of view, an USB Pendrive is a “block device”, i.e. a device with a certain amount of addressed blocks, and it behaves EXACTLY as Hard Disk Drive.
Hard Disk block access was traditionally achieved by means of the so called CHS address (each block had an address composed by its Cylinder/Head/Sector address).
See here for reference:
Later development (and increasing size of Hard Disks) led to use another system, ECHS:
Later again, development (and increasing size of Hard Disks) led to use another system, called LBA (Large Block Address), backwards compatible, that uses the absolute number of the sector to find it:
And here find a comparison of the three:
On ALL modern motherboards, LBA is used, so that we might face the different BIOS caveat here:
This can lead to a bootable Pendrive that is PERFECTLY working on one PC and is NOT on another!
I made this digression to introduce the fact that there is NO GEOMETRY set in the MBR itself, and it can vary on different BIOS interpretations, moreover as it has to be represented nonetheless as a CHS value in MBR, the REPORTED size by BIOS could be different from the actual one, as the three CHS values are multiplied by themselves, and given a certain geometry, total size needs to be a multiple of the three said values, and this can produce either “surplus” or missing sectors. (we will see later that actually there is some geometry info, NOT in the MBR, but in the BOOTRECORD)
Let’s see the example of my KingMAX 128 Mb:
Formatting it with the HP utility as FAT16, result window says:
130,766,848 bytes total disk space
2,048 bytes in each allocation unit
63,851 total allocation units on disk
Now the result is correct:
2,048 * 63,851=130,766,848
Now use this fine FREEWARE utility to see the partition table and geometry:
It says Cyls 15 Heads 255 Sectors 63
As each sector is 512 bytes, we have:
15 * 255 * 63= 240,975 sectors * 512= 123,379,200
Quite different, isn’t it?
The total number of sectors in Partition one is 255,937 sectors,
255,937 sectors * 512= 131,039,774
adding the 63 “hidden” sectors of MBR and track 0, we get 256,000 sectors,
256,000 sectors * 512= 131,072,000
(probably the 128 Mb the Pendrive is called for)
If I select the Pendrive in Windows Explorer, right-click/properties, I get
130,766,848 bytes, that corresponds to the 255,937 sectors seen above, if you take into account space reserved for FAT tables, BOOTRECORD etc.
Let’s start from this one:
255,937 /63 Sectors= 4,062.4920…. C * H which is obviously not possible
255,937 /255 Heads=1,003.5674…. C * S which again is not possible
255,937 /15 Cylinders=17,062.4666… H * S which again is not possible
Let’s try again from the “FULL” one:
256,000 /63 Sectors= 4,063.4920…. C * H which is obviously not possible
256,000 /255 Heads=1,003.9215…. C * S which again is not possible
256,000 /15 Cylinders=17,066.6666… H * S which again is not possible
The Pendrive can boot allright, but if you try to access it as “Physical Drive”, the total size will be 123,379,200.
This evidently means that both BIOS booting code AND direct access from Win2k use the CHS convention, whilst “Normal” drive operation under Win2k uses LBA.
Let’s try to use the DELL utility, results in MBR, through Beeblebrox, are:
Startsec (=number of previous sectors ) 32
Let’s recall how the Numsectors should be calculated:
so in this example:
So the mathematics is correct, but we have a
total capacity of 250,624*512=128,319,488
and a partition size of
Capacity reported by Properties of the drive is 128,034,816, correct if you take into account space reserved for FAT tables, BOOTRECORD etc.
The key formatted as above WILL NOT boot on my system.
Now let’s manually edit the MBR and see what happens:
(Note: you can use the said Beeblebrox to edit the MBR, but you will need another utility to edit the BOOTSECTOR, there is this one from Powerquest, PTEDIT32, very similar to Beeblebrox, which you can find here:
please note that the Licensing status of this app is UNKNOWN,
or use a freely available disk editor like Microsoft’s own DiskProbe, find it together with some more ResKit tools here:
In the links section of this article you’ll find links to more FREEWARE disk/hex editors)
Every utility has a slightly different way to name the same data, I use a mix of the “name tags” different programs use, as I find it easier, please refer to the following table for Cross-reference:
|My tags||Beeblebrox||Ptedit32||DiskProbe||Ranish PM|
|Boot||Boot||Boot||Boot Indicator||Boot flag|
|bHead||BHd||Starting Head||Starting Head||Starting Head|
|bSect||BSec||Starting Sector||Starting Sector||Starting Sect|
|bCyl||BCyl||Starting Cyl||Starting Cylinder||Starting Cyl|
|eHead||EHd||Ending Head||Ending Head||Ending Head|
|eSect||ESec||Ending Sector||Ending Sector||Ending Sect|
|eCyl||ECyl||Ending Cyl||Ending Head||Ending Cyl|
|SectBefore||StartSector||Sectors Before||Relative Sectors||Starting:|
Here is the same data, with offsets for first parttition table entry, as seen in the FREEWARE hex editor Hackman and in the COMMERCIAL one WinHex:
|My tags||Offset (hex)||Offset (dec)||Length Bytes||Hackman||Winhex|
|Boot||01BE||446||1||BIOS Handle||active partition|
|bHead||01BF||447||1||First head||Start head|
|bSect||01C0||448||2||Starting sector and Cylinder||Start sector|
|bCyl||01C0||448||Starting sector and Cylinder||Start cylinder|
|Ptype||01C2||450||1||Partition type||Partition type indicator (hex)|
|eHead||01C3||451||1||Last head||End head|
|eSect||01C4||452||2||Last sector and cylinder||End sector|
|eCyl||01C4||452||Last sector and cylinder||End cylinder|
|SectBefore||01C6||454||4||Sectors until partition||Sectors preceding partition 1|
|TotSectors||01CA||458||4||Number of sectors||Sectors in partition 1|
Now we have in the MBR the following situation:
(remember that Win2k reports Disk Geometry as Cyls 15, Heads 255, Sectors 63)
Let us correct it:
The BOOTRECORD is recorded in sector 32, i.e. at address 32*512=16,384, but the CHS geometry seen by BIOS tries to find it at C0 H1 S1, i.e. 1*63*512=32,256.
Change the Bhd/Bsec as follows
And try to boot from the Pendrive, surely enough you will be greeted by the FREEDOS prompt!
FreeDOS, being a bit “smarter” than other Operating system will complain with something like:
Found suspect partition with geometry 978*7*32, using calculated geometry 248*10*10 (yet another different one!), but will boot all the same.
Now reboot and from Win2k copy to the Pendrive Ranish Partition Manager, which you can find here:
For this particular operation I have used both the “stable” 2.40 version and the “Beta 2.43” and “Beta-Beta 2.44” versions with no problem, the 2.37 won’t work on the Pendrive, but when we will get to more advanced tricks, ONLY the STABLE 2.40 should be used.
While you are at it, download XOSL also, you might need it later if you decide to follow my advice.
From the downloaded .zip file, extract to the Pendrive part.exe from the 2.40 version and save it as p240.exe, copy also the CWSDPMI.EXE.
Reboot and when (if) the FreeDOS prompt appears start p240.exe.
It will find yet another geometry C253*H16*S63.
Let’s change again some data:
(with the partition highlighted press F4 to see the Number of Sectors and Total Sectors data)
Now press enter and use the arrow down key to edit the Number of Sectors, setting them to 254992 Ranish PM will complain that bootsector data is invalid, you can ignore it for the moment, press ESC and save the MBR.
Try to reboot, should boot OK, WITHOUT the “suspect partition geometry” warning.
So we have seen how an incorrect fdisking/formatting can be manually corrected.
Now let us re-run the HP utility, formatting the stick as FAT16 bootable, (you will need to provide the following files from WIN98SE:
(which you can find on any “bootdisk” you can download from the net)
You might need also this two files:
depending on which bootdisk you got.
Though it “should” be possible to load on the Pendrive DOS 6.22, when I tried to do so I had some problems, so I do not recommend it.
As seen before, FREEDOS works on a Pendrive, there are reports that it is sometime tricky to get it to work, actually I used it without any problem, and on the contrary it has some nice features that Microsoft DOS 7.1 (a.k.a. Windows 98 SE) misses.
Let’s have a look a the Partition table with Beeblebrox:
Cyls 15 Heads 255 Sectors 63
We have the same difference seen above:
Total size (from disk geometry):
Total size CHS:
Total size LBA:
If we boot from the Pendrive and use on it Ranish Partition Manager and change the ending sector like this:
We have a somewhat more satisfactory result:
Total size CHS:
But it remains unsatisfactory if you want to have the MAXIMUM compatibility.
I know I already made you reboot an infinite amount of times, so let us start to find a way to experiment WITHOUT rebooting.
The solution is QEMU:
Download from here:
the latest win32 install,
then go here:
and get the nice qGUI
We need a good FREEDOS bootdisk, I got mine here:
We still need some more utilities:
Ken Kato’s Virtual Disk Driver:
and of course my pseudo-GUI for it:
Hackman Template Editor:
and get MBRFIX:
and my pseudo GUI for it
(you will need to log in to get those)
There is another more complete GUI for MBRFIX here:
(again you will need to log in to get this)
that makes use of MBRWizard 1.52 too, which you can get here:
Let’s start from clean:
- Use Roadkil Diskimg to make an image file of your pendrive.
- Make a copy of it (just in case)
- Open the first copy with Frhed, select all and FILL it with 00
- Now you have a completely BLANK image of the Pendrive.
- Mount it with VDK using the ezMount option, the image will be mounted, but, as there is NO partition in it, NO letter drive will have been assigned.
- Open Beeblebrox, you will find Physical Disk 0 (THIS IS YOUR HARD DISK, DO NOT FIDDLE WITH IT) and Physical Disk 1 (the Pendrive, unless you have more than one hard disk, this is the image, if you have more than one hard disk, numbers must be incremented); open it with PTEDIT32, you will have Drive 1 (the HD) and Drive 2 (the Pendrive); ALWAYS PAY ATTENTION AS DIFFERENT UTILITIES “CALL” THE SAME OBJECT WITH DIFFERENT NAMES.
The geometry shown will be 125*64*32, let’s make a table of these results:
|Dos booted from key/RPM||Geometry||253||16||63||255,024||130,572,288||127,512||125|
|The CHS is fractional||253||16||63||255,024||130,572,288|
- Now it is quite obvious that if there are so many different ways to read the same data, it is PROBABLE that different Motherboard manufacturers have implemented different ways to access the data, if the Motherboard AND the operating system you are going to install on the Pendrive BOTH use LBA, you have no problem at all, but if either use CHS, something will not work.
- Now we need to put an MBR on the image, best option is using MBRfix with one of the GUI’s, but you can copy with Frhed or any other hex editor sector 0 of any Hard Disk you have and paste it on sector 0 of the image.
- Now re-open it with either Beeblebrox or PTEDIT32, if you used MBRfix you will have a blank partition table, otherwise you have to blan it manually with Beeblebrox/PTEDIT32, what we want to do is to find a start (BOOT) sector address for the FIRST partition that can be reached both with LBA and CHS and with ALL the geometries seen above, the only one that fulfills this is Cyl 0 Head 0, we could start as low as Sector 2 but as some utilities use the first few hidden sectors for their data, including the “normal” FAT32, best option is to get the last addressable sector with geometry 125*64*32, BEFORE “switching” to Cyl 0 Head 1, i.e. Cyl 0 Head 0 Sector 32, this way even if cylinders, heads and sectors number differs, the result is GUARANTEED, let us set the size of the partition , so that the entire partition can be accessed again BOTH as LBA and CHS on ALL geometries.
The limit to obtain the said Maximum compatibility is the maximum size seen from Win2k, i.e. 123,379,200, corresponding to 123,379,200/512=240,975 sectors, which represented in a 125*64*32 leads to Cyl
Let’s make it:
- Now unmount the image from VDK via ezUnmount and re-mount it via ezMount, this time a letter will be assigned to the partition we just created.
- Open the drive letter with Explorer and it will ask you to format it, format it as FAT(16)
- Now you will need to setup Qemu you will need to mount from VDK via ezMount (yes VDK can mount floppy images too) a copy of the freedos floppy disk image or a Windows 98 boot disk, to which you need to add:
– Ranish Partition Manager 2.40
– Xfdisk 0.93
– Zeleps Partition Resizer
Unmount the floppy image and start Qemu with the floppy disk image as floppy, the disk image as Primary master, setting it to boot from A
Run Ranish Partition Manager, and you will see that geometry is read this time as Cyl 253*Hd 16*Sect 63, partition table is now seen like this (press F4 to see TotSectors):
- Using Ranish, create a partition on the remaining of the drive:
(last line is seen ONLY in Ranish Partition Manager)
- Now you can re-boot Qemu and do:
sys a: c:
sys a: d:
While you are at it, copy from the floppy the files we will need later:
copy boot*.* c:
copy NT*.* c:
copy p240.exe c:
copy cwsdpmi.exe c:
copy xfdisk.* c:
now change drive to c: and issue:
copy *.* d:
answer NO if you are required to overwrite Command.com
- Close Qemu and reboot it setting to boot from C, you will be greeted (hopefully) by the DOS/FREEDOS prompt!
- Now you have on the image TWO primary partitions, this, as the partitioning scheme seen above, is NOT standard, so it will behave differently in different environments, until we are in the “virtual” everything is OK:
– QEmu will boot
– VDK will mount (manually) BOTH volumes
Once the image is transferred to the Pendrive:
– it WON’T boot, as the BIOS expects to find just one primary partition
– Win2k will ONLY assign a letter to the FIRST partition on the drive
- To make the thingy bootable from BIOS, we need an utility to HIDE the other primary partition at boot, you can use XFDISK, which is very easy to setup, but in the configuration proposed gave incorrect drive size. With all the other partitioning schemes I tried it worked flawlessly, still looking into the matter.You can setup the boot manager that comes together with Ranish, but it is a little tricky and it seems like I am not able to find a quick way to hide/unhide partitions at startup.
The VERY good XOSL boot manager gave me some errors under Qemu, but it worked for me on the real Pendrive.
Luckily enough, I found another nice very simple bootmanager called PARTITA, which you can find here:
the good thing it has is that it fits ENTIRELY in the MBR, even it is a bit “unfriendly” at setup. As it needs to find BOTH partitions marked as visible BEFORE installing, it can be done EXCLUSIVELY from within Qemu, here are the steps involved (copy PARTITA.EXE to the disk image mounted with VDK, then unmount it and run it from Qemu):
partita <-shows partition info->
partita /1:BIG <-names the first partition “BIG”->
partita /2:small <-names the second partition “small”->
partita /d:1 <-sets partition #1 as default->
partita /t:30 <-sets the timeout before default partitio is chosen->
partita /1-234 <-when partition #1 is chosen, 2,3 and 4 are hidden->
partita /2-134 <-when partition #2 is chosen, 1,3 and 4 are hidden->
- Re-boot QEmu to test that everything is ok, you should be able to boot from BOTH partitions, selecting which at boot time.
- Now we can transfer the modified disk image back to the Pedrive, via Roadkil’s DiskImg
- Cross your fingers and re-boot computer
- If everything works OK, let’s make the boot entries for the NTLDR:Boot in one of the two partitions and issue the command:
then, depending on what you had in your BOOT.INI, use
BOOTPART remove #
(with the appropriate number of the entry to be deleted), then run
BOOTPART WIN98 C:\BOOTSECT.W98 “Windows 98”
BOOTPART WINNT BOOT:C:
- Re-boot and test it, if it works, reboot in the second partition and repeat the above, DO NOT copy BOOTSECT.W98 from one partition to the other. If you used FREEDOS, you can copy with FRHED sector 0 of each volume mounted with VDK and save it EACH in the proper volume as BOOTSECT.FDS and edit manually accordingly each BOOT.INI.
- You can later add an arcpath entry like
with any editor and copy to the drive the relevant XP/2k/BartPE files
As said before this is the first draft, the above are just examples, I hope it is enough to get you started.
That’s it for the moment, next update will be referring to:
using the other utilities listed before
Using this app:
which I recently found and that needs some more experimenting.