FloppyBuilder evolution

Anything related to the tools Tap2Wav, Tap2CD, Tap2Dsk, Sedoric Disc Manager, Tape Header Creator, WriteDsk, and generaly speaking tools related to the management of Oric data files and devices.
User avatar
Dbug
Site Admin
Posts: 4437
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

FloppyBuilder evolution

Post by Dbug »

I created this new topic to discuss the changes in the new versions, I'm looking for feedback, so if something seems to you to be a bad idea then by all means express yourself :)

I've been discussing with Chema and Iss about their use of the system, and so far most of the changes have been in the actual loader code, not the FloppyBuilder itself, but for the next evolution of the system the program itself will have to be modified.

One of the first things I'd like to mention is that I'm not planning to be backward compatible with the previous versions, so upgrading to a new version may not work well. That being said, the modifications should be relatively easy to apply, and I plan to add a new instruction to define the format version the description file was created for, and FloppyBuilder will output warnings or errors if it knows that it is not compatible with the version you targeted, so at least it will not silently fail.

So, here is a first list of what I was thinking in doing:

- Add a "FormatVersion" instruction, followed by the FloppyBuilder version. DONE
If you have in your script "FormatVersion 0.19" and you try to use it in version 0.20 of the FloppyBuilder, it will tell you that you need to fix your script.

- Add a "WriteLoader" instruction. DONE
it will behave as the existing AddFile instruction except it will not add the loader in the list of files, and it will automatically generate the defines with the location of the loader in memory, as well as the sector and track positions.

- Remove the loading address from the "AddFile" instruction. DONE
After having used the system for quite a while now, I realize that the loading code always knew where the file was going to be loaded, so ultimately we are just wasting space in the file directory information for nothing.

- Add new macro magic values. DONE
We have {FileIndex} and {FileSize}, we should probably also have {FileTrack}, {FileSector}, {FileIsCompressed}

- Add some parameter to auto generate defines based on file names. PENDING
If you have a "AddFile MySuperFont.hir" it could automatically generate #define LOADER_MYSUPERFONT_HIR_INDEX 23 using the name of the file to generate the define

- Allow two different formats for the file directory table. PENDING
Right now we have individual arrays for sector, track, size, address, but in some cases it would be more practical to have the data interleaved

- Modify the detection of compressed/uncompressed files. DONE
Right now the loader checks if the compressed size of a file is equal to the uncompressed size and uses that to decide which loading code to use.
I suggest removing one of the two size information, and store the "compressed" flag as the high bit of the Sector number, that will not make the code any more complex, but that's two bytes less per entry of the directory.

- Possibility to add defines at the command line. (from Iss) DONE
This would allow the possibility to generate different DSK files using the same script, by having some defines passed on the command line and exported in the header file (precedence of the command line version if there's already a define with that value in the script)

- Defines should appear *after* the AddFile commands. (from Iss) DONE
For coherence, all the AddDefines should refer to the previous AddFile instead of being before.

- Jasmin support. PENDING
With a bit of work the code can be made compatible to both Jasmin and Microdisc without any size or performance cost.

- Allow the loading of uncompressed files on non page size multiple. DONE
Right now compressed files can be loaded in any buffer that is large enough to contain them, but uncompressed files overwrite memory by multiples of 256 bytes, which is unpractical.

- Loading/Saving indicators support. PENDING
Right now if you want to show an animation while loading (spinning disk, rasters, message), you have to manually edit the api source code. Ideally it should just be a function to call.

How does that sound?
User avatar
Dbug
Site Admin
Posts: 4437
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Re: FloppyBuilder evolution

Post by Dbug »

So, some feedback on what I managed to do so far:

Code: Select all

0.20 - 2016/01/19
- Added a 'FormatVersion' command to help handle the lack of backward compatibility
- Added a 'WriteLoader' command to simplify the handling of loader specific parameters (the loader cannot be compressed, should not be in the directories, etc...)
- Three new defines are automatically created: FLOPPY_LOADER_TRACK, FLOPPY_LOADER_SECTOR and FLOPPY_LOADER_ADDRESS. They are designed to be used by the boot sectors to help load the loader.
- Added a new set of macro variables: {FileTrack}, {FileSector}, {FileSize} and {FileSizeCompressed}
I'm working on the Weekend In Oslo Game Jam of last year as a test program for the improvements, and the main changes so far are that I managed to get the whole list of files out of the loader itself (it can now be integrated in the main program and changed on the fly).

Of course that means the entire loading API has changed, as well as the memory layout and the initializations, but now I do have a loader.asm that fits in less than 512 bytes, and the position on disk as well as the loading address in memory are all defined by the description script.

The following lines:

Code: Select all

SetPosition 0 4
WriteLoader ..\build\files\loader.o $fe00 			; Sector 4
auto generate the following output in the generated header file:

Code: Select all

#define FLOPPY_LOADER_TRACK 0   // Track where the loader is stored
#define FLOPPY_LOADER_SECTOR 4   // Sector where the loader is stored
#define FLOPPY_LOADER_ADDRESS 65024   // Address where the loader is loaded on boot $fe00
I'm still working on the whole system, so don't expect it to stay in 512 bytes, it will grow-up again, if only to get the Jasmin compatibility, the save code, and the partial loading of files :)

(Can still dream though, would be cool :p)
User avatar
Chema
Game master
Posts: 3013
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: FloppyBuilder evolution

Post by Chema »

Really nice Dbug!!!
User avatar
iss
Wing Commander
Posts: 1637
Joined: Sat Apr 03, 2010 5:43 pm
Location: Bulgaria
Contact:

Re: FloppyBuilder evolution

Post by iss »

Very good news! I'm looking forward to test the new features.
User avatar
iss
Wing Commander
Posts: 1637
Joined: Sat Apr 03, 2010 5:43 pm
Location: Bulgaria
Contact:

Re: FloppyBuilder evolution

Post by iss »

Incredible progress, Dbug!
User avatar
Dbug
Site Admin
Posts: 4437
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Re: FloppyBuilder evolution

Post by Dbug »

So here is a candidate version of the 1.0 FloppyBuilder, with last year's Global Game Jam project modified to use it.
GlobalGameJam2015.zip
GlobalGameJam project + FloppyBuilder 1.0
(284.91 KiB) Downloaded 611 times
It's not the final version, but I thought that giving some early information regarding what the system will look like would be nice.

The main differences are:
- Format changes in the description files, mostly the requirement for the FormatVersion instruction, the WriteLoader instead of AddFile, the removal of the address parameter after AddFile, the AddDefine {FileIndex} being placed after the AddFile, and the new {File*} parameters

- "disc_info.h" is gone, all the necessary informations have been moved in the respective files (boot sectors and loaders)

- The loader does not include the directory table anymore, and is made of two sections, the loading of the main program is hardcoded in the variables at the end of the file, also the first part of the loader is designed to just contain initialization stuff (in this case setting up the VIA and the Jasmin parameters) and can be overwritten after that. (As long as nothing writes in memory after the label _LoaderResidentStart you should be fine).

- The assembly address of the loader are now all defined in the description file: LOADER_BASE_ZERO_PAGE, LOADER_SECTOR_BUFFER and FLOPPY_LOADER_ADDRESS are exported to the generated header file and used by the loader source file.

- The directory file is now included in loader_api.h and .s, which now only has the LoadFileAt version (LoadFile is gone, you have to specify the address).

The next two big changes will be to provide a way to get the loading/saving indicators, and generation of directory files that can be loaded on the fly.

EDIT: Just noticed that the version I included still generates the FileStoredSizeLow/FileStoredSizeHigh tables, they are not necessary anymore and are removed in my current version
User avatar
coco.oric
Squad Leader
Posts: 720
Joined: Tue Aug 11, 2009 9:50 am
Location: North of France
Contact:

Re: FloppyBuilder evolution

Post by coco.oric »

Very impressive,

I'll wait this improvements before trying it
coco.oric as DidierV, CEO Member
Historic owner of Oric, Apple II, Atari ST, Amiga
User avatar
Dbug
Site Admin
Posts: 4437
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Re: FloppyBuilder evolution

Post by Dbug »

coco.oric wrote:Very impressive,

I'll wait this improvements before trying it
Actually the final version will be very close to that one, so I'd rather have as many people as possible trying it so I can collect all the feedback and fix the stuff before doing it an official release.

This version breaks the compatibility with the previous version (thus the 1.0 label), I'd rather not break the compability again :)
User avatar
iss
Wing Commander
Posts: 1637
Joined: Sat Apr 03, 2010 5:43 pm
Location: Bulgaria
Contact:

Re: FloppyBuilder evolution

Post by iss »

Attached files are looking good for me (this means I still understand how they work ;)).
Else, sector_2-microdisc.asm has the relocator bug (missing '#'), but maybe you just didn't use your last versions of sources.
About load/save indicators, I think it's enough to export in loader_api just the pointers to show/hide functions, which by default point to RTS. Additionally one simple example in OSDK can be included to show how to use these indicator functions. This will provoke developers to show their creativity :).
Godzil
Squad Leader
Posts: 774
Joined: Sat May 21, 2011 7:21 pm
Location: Between UK and France
Contact:

Re: FloppyBuilder evolution

Post by Godzil »

As I'm working on floppy drive currently, I found a potential mistake in DBug microdisc bootsector, and I have a potential warning (even if I'm sure you are already aware of that)

First, I don't understand why you set back the microdisc 314 register with that value at the end of the bootsector, changing the bit 3 and 2 to both 0 are not "valid", they should be not changed and stay as 10 for the "normal" double density value.

Bit 3 drive the FDC to say to work in double density mode, and both of them drive a chip that decipher part of the physical information from the floppy and select a clock divisor for that chip. 00 is not an invalid combination; but if not correct for normal floppy read.

Also a bit of warning, even if the FDC1770 and FDC1793 instruction set look pretty similar, in fact bit7 to bit 4 are the same for both, there are some slight and important differences in bit 3 to bit 0 between the two. And of course the microdisc use the FDC1793, the Jasmin the FDC1770, so be careful with that, using the same commands of the microdisc with the jasmin may cause issues.

As I'm here, some comments about a bit comment in Floppy.cpp:

For 18 (19?) Sector mode, the gap1 byte value is bellow the minimal value set on the FDC1793 documentation, I don't know where they had that nice idea to lower it that much. (reliability is not related to the fact that the track header is not there as the 1793 does not need it)

Also:

Code: Select all

  /*
  Format of a track:
  6400 bytes in total
  - gap1: 72/12 bytes at the start of the track (with zeroes)
  Then for each sector:
  - ?:             4 bytes (A1 A1 A1 FE)
  - track number:  1 byte (0-40-80...)
  - side number:   1 byte (0-1)
  - sector number: 1 byte (1-18-19)
  - one:           1 byte (1)
  - crc:           2 bytes (crc of the 8 previous bytes)
  - gap2:          34 bytes (22xAE , 12x00)
  - ?:             4 bytes (A1 A1 A1 FB)
  - data:          256 bytes
  - crc:           2 bytes (crc of the 256+4 previous bytes)
  - gap3:          50/46 bytes
  */
The first "?" 4 bytes are 3 sync bytes (A1) to tell that the next byte is the type of the following data and FE which is in that case the AddressMark tag, meaning that the next bytes describe the current sector
for the second "?", same thing and FB is to say "the next chunk are the data"

You also find FC which is for the track header. The one I don't know yet is FD, but I think it maybe related to deleted sector, or marked sectors marked as bad.

and last thing:
Lors de l'élaboration du tampon de formatage SEDORIC, les octets #C2 sont remplacés par des octets
#F6, les octets #A1 sont remplacés par des octets #F5 et chaque paire de 2 octets [CRC CRC] et
remplacée par un octet #F7. Comme on le voit, nombre de variantes sont utilisées, sauf la zone 22 [#4E],
12 [#00], 3 [#A1] qui est strictement obligatoire.
The 0x4E*22, 0x00*12, 0xA1*3, 0xFB is not the only required, the 00*12, 0xA1*3, 0xFE and information about track, side, sector number, size and CRC is also compulsory, without the FDC will not be able to find the requested sector.

And the transformation is not relevant in FloppyBuilder case as it's to make the data to be sent to the FDC.
With the write track command, values from F5 to FF have a special meaning for the FDC as for exemple, writing C2 or A1 for the index marker is not enough as it may have some effect on how the signal to the floppy is generated, and that's a missing information in our current "MFM" format, we only have the data and not the real thing stored on the floppy. Not that's it's really important by the way, but we could consider that on some platform some people used some of that "out of band" information for copy protection. It's not the cas on the oric, so it's not a big deal.

But I think that this information is not interesting for FloppyBuilder as what we write is the final data, not the data send to the FDC, unless we make a binary that is going to be send to a FDC to write a real floppy.

If you want more information, look at that datasheet page 14: http://bitsavers.informatik.uni-stuttga ... _May80.pdf

By the way I more recommend this version of the FD1793 datasheet as I find it to have a better layout, and more useful than the one found here on DF
User avatar
Dbug
Site Admin
Posts: 4437
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Re: FloppyBuilder evolution

Post by Dbug »

Hi,
an important thing to consider is that I did not create all that from thing air:
- The original microdisc code comes from our ealier demos, was written by Jede, and it worked just fine. I did not try to change anything regarding the order of operations or timings, it may contain errors or useless operations or sub-optimal sequences of commands
- The gap/disk structure information, same thing: It comes from Fabrice tools, I reused his tables of values and generic logic.

My way of working is simple: If it works, I don't try to fix it :)
Now, I'm all open for improvements - as long as we are sure that things still work on real machines.
Godzil wrote:First, I don't understand why you set back the microdisc 314 register with that value at the end of the bootsector, changing the bit 3 and 2 to both 0 are not "valid", they should be not changed and stay as 10 for the "normal" double density value.
No idea, that was in the original code. Maybe we can just remove the instruction.
Godzil wrote:Bit 3 drive the FDC to say to work in double density mode, and both of them drive a chip that decipher part of the physical information from the floppy and select a clock divisor for that chip. 00 is not an invalid combination; but if not correct for normal floppy read.
As long as we don't try to access the disk, I don't think that causes any problem; but sure, using correct values is probably a good idea.
Godzil wrote:For 18 (19?) Sector mode, the gap1 byte value is bellow the minimal value set on the FDC1793 documentation, I don't know where they had that nice idea to lower it that much. (reliability is not related to the fact that the track header is not there as the 1793 does not need it)
All that is things I collected from Fabrice tools and some from some CEO related documentation and articles.
Godzil
Squad Leader
Posts: 774
Joined: Sat May 21, 2011 7:21 pm
Location: Between UK and France
Contact:

Re: FloppyBuilder evolution

Post by Godzil »

Don't get me wrong, I know that you are not creating all of that from thin air, and you reuse code and document that has been proved to work!

That wasn't my goal! I'm just looking at schematics, datasheet and existing code to see/understand how our beloved microdisc and Jasmin works for the Cumulus Rev2 project. And when I found something that sound strange/wrong, I just ask myself question, and if I know that something could be interesting, I share, or ask question. I shared for exemple the warning about the differences in the commands between the 1770 and the 1793 so, we need to be carefull on common code for the Jasmin/Microdisc as there are slight differences.

And ask question about, for exemple, the register set at the end of the microdisc init code because I don't know/understand why it has been done this way as the value does not mean anything that seems valid, at least in our context. 00 is a valid value and will not make either the cumulus or the microdisc to explode, but trying to access a floppy with that setting will just not work, but I suppose that the next part of the code will set it back to the normal value, so I'm not that worried, it's just a potential bug.
User avatar
Chema
Game master
Posts: 3013
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: FloppyBuilder evolution

Post by Chema »

Yesterday I set up an old laptop with Win95 to write a disk and test my new game on the Oric using a real floppy drive. My intention was to see if there was a clear difference in performance when the actual mechanic parts were being used.

First of all, let me tell you there is not a big difference, and the performance is quite adequate. Which is good news :)

But I found it very hard to make the disk boot. I am using the first version of Floppy Builder, not the newer evolution, so maybe there was a hidden problem that has been solved. I have to upgrade, but it was working and I preferred to wait until I really needed the new features.

The symptoms are: I press the reset on the microdisc and 1 out of, say, 10 times the disk boots, the game loads and everything seems to go well. The rest of the times the drive starts, the screen is switched to black (hires, I think) and it hangs.

As I have the purple bar at the bottom whenever a sector is read, I am pretty sure the problem is in the microdisc bootloader code (sector_2-microdisc.asm) in the read_sectors_loop code.

Also as it works perfectly with Cumulus and emulated Orics, I think it could be related with read errors or timings on the real FDD.

As a side note I have tested with several different disks (some were working). Other programs using SEDORIC or ORIC DOS (1337, the sedoric master disk, Space:1999, Murder on the Altantic,...) work perfectly and flawlessly.

I may also add that sometimes the sector read seems to take longer than in the emulated hardware (the purple bar stays on for some time). Maybe some retries or hardware taking longer to find the sector...

Any clues?
User avatar
Dbug
Site Admin
Posts: 4437
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Re: FloppyBuilder evolution

Post by Dbug »

Could try on the same machine the older Defence Force demos, such as Barbitoric or Quintessential ?
User avatar
Chema
Game master
Posts: 3013
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: FloppyBuilder evolution

Post by Chema »

I am pretty sure both worked (although the one with the digitized sound sounded horrible). This is the first time I find this thing occurring, Dbug.

I will test again, just in case my memory fails...

The sequence is:

Press reset - "insert system disk" message appears - FDD light flashes, motor starts - screen goes black (hires I think) - everything stops working - FDD led goes off

I think that the FDD motor stops just before the screen goes black, but I can't be sure.

Make 5-10 attempts and it works. The loading continues from the black screen. FDD keeps on working and the magenta bar flashes (as sectors are read). I read data many times from that point and always successfully.

I only once had an error when loading data after the boot was successful. And I could not reproduce it again. I think the magenta bar kept on, which means the sector was never read, and the FDD stopped.
Post Reply