FloppyBuilder evolution

[Chema]

Dbug, are you aware of any bug in the unpacking routine in the first version of FloppyBuilder? I am facing a strange bug which prevents things to load that appears and disappears as I enter/edit new code or add extra bytes. The bug disappears if I remove the FilePack option.

I know I have to move to the latest version, but I need time and energy to migrate.. Could you provide a quick and dirty guide about how to do it?

[Dbug]

Not that I'm aware, but it's always possible

If you could write a minimal test case that exhibits the problem, would be helpful.

[Chema]

Not that I'm aware, but it's always possible

If you could write a minimal test case that exhibits the problem, would be helpful.

That will be difficult, Dbug, but I will try to see if I can narrow the issue a bit more. The thing is that I have to add extra bytes to the main program (in a certain range, no more no less ) so the loading of another part from disk works (part which has not changed). That is quite strange unless the loader code is disturbed somehow or the main program code has been decompressed with some kind of error.

Now that I am writing this... I could try to examine the first 16K of a dump both loaded with compression and without it... they should be the same....

[Chema]

Comparing dumps showed nothing strange. But I found something else.

The file which fails loading correctly is not compressed (it is small). It is number 2 in the data arrays, just after the main program which is either compressed or not depending on the floppy builder option. This main program is number 1 (number 0 is the loader).

When compressing and with no extra bytes added the generated floppy_description.h file the sector info is:
FileStartSector .byt 4,7,7,8,9,10,11,12,4,10,12,4,6,7,8,10,13,14,9,7,1,8,16,7,2,11

But the debugger shows that the stored information for the start sector is :

04 07 06 07 08 09 0A 0B

The thing is that if I add the extra bytes to the main program, the compressed size is (believe it or not) smaller (12800 vs 12804) enough to make the header file information now:
FileStartSector .byt 4,7,6,7,8,9,10,11,3,9,11,3,5,6,7,9,12,13,8,6,17,7,15,6,1,10

It is quite strange because I think this information ends up in the loader code with an #include directive, so either it is using a different (older?) header file or I have no idea what is going on. In addition I also include the same file for my directory (yeah, something that should be changed I know) and it is also wrong (in the debugger both memory areas show the same information).

I tried deleting the header file and the build folder before compiling, with no success.

I am quite sure that it is reading the end of the previous file, as it has some bytes of information and the rest are all zeros.

So if this all makes sense then it is a good hint towards the bug. Else it is elsewhere and I simply made some of the tests wrong.

P.S. As a side note if I remove compression the sector information is correct in the file and in the debugger, although it is (of course) different from the one above.

Edit 2: 12800/256=50 sectors exactly.

[Dbug]

Hmm.

FloppyBuilder when asked to compress files tries to compress them, but if the compressed size is equal or largest to the non compressed one, then the file is stored uncompressed.

Code: Select all

  std::vector<unsigned char> compressedBuffer;
  if (m_CompressionMode==e_CompressionFilepack)
  {
    // So the user requested FilePack compression.
    // Great, we can do that.
    compressedBuffer.resize(fileSize);
    gLZ77_XorMask=0;
    size_t compressedFileSize=LZ77_Compress(fileData,compressedBuffer.data(),fileSize);
    if (compressedFileSize<fileSize)
    {
      // We actually did manage to compress the data
      fileData=compressedBuffer.data();
      fileSize=compressedFileSize;

      fileEntry.m_CompressionMode=e_CompressionFilepack;
      fileEntry.m_StoredFileSize=compressedFileSize;
      fileEntry.m_SectorCount=(fileSize+255)/256;
    }
  }

if it did not manage to compress then it uses the default parameters:

Code: Select all

  fileEntry.m_StoredFileSize=fileSize;
  fileEntry.m_FinalFileSize   =fileSize;
  fileEntry.m_SectorCount=(fileSize+255)/256;
  fileEntry.m_FilePath   =fileName;
  fileEntry.m_CompressionMode=e_CompressionNone;

The thing is that if I add the extra bytes to the main program, the compressed size is (believe it or not) smaller (12800 vs 12804)

Well, LZ77 is based on pattern matching and repetition, so adding bytes in some places may just may make this place match another one.

When compressing and with no extra bytes added the generated floppy_description.h file the sector info is:
FileStartSector .byt 4,7,7,8,9,10,11,12,4,10,12,4,6,7,8,10,13,14,9,7,1,8,16,7,2,11
But the debugger shows that the stored information for the start sector is :
04 07 06 07 08 09 0A 0B

Just one question: Are you using the FilePack as in my demos, or did you integrate it in your own build script? If you integrated in your build script, did you do the correct number of loops so all the programs that include FloppyBuilder generated files actually get recompiled/reassembled with the correct data?

Basically the reason for the multiple passes is that in order to be able to allocate sectors efficiently and without waste, the program needs to know the size of each file, and since some of the files include some FloppyBuilder information (which is dependent of the build script) it needs to build the whole project multiple time.

If you don't do it the right number of time, you get out of sync information.

What has to be rebuilt multiple time is only stuff that includes generated data, the rest is "binary stable" and will not change from one build to another

[Chema]

I knew the code is good enough as to detect if compressing was worth it (thought I'd say maybe check not only if the size is less but maybe have some kind of margin, such as if the compressed data uses less sectors).

It must be something related to the chicken-and-egg problem. I am using my own build batch file, but based on yours, so calling FloppyBuilder twice (I have tried making 3 and 4 iterations with no different results). Here is my code:

Code: Select all

:: Call FloppyBuilder once to create loader.cod
%osdk%\bin\FloppyBuilder init floppybuilderscript.txt


set FLOPPYPASS=-

:Loop
echo %FLOPPYPASS%
:: Call XA to rebuild the loader
ECHO.
ECHO Assembling bootsectors
%osdk%\bin\xa -DASSEMBLER=XA sector_1-jasmin.asm -o ..\build\files\sector_1-jasmin.o
IF ERRORLEVEL 1 GOTO Error
%osdk%\bin\xa -DASSEMBLER=XA sector_2-microdisc.asm -o ..\build\files\sector_2-microdisc.o
IF ERRORLEVEL 1 GOTO Error
%osdk%\bin\xa -DASSEMBLER=XA sector_3.asm -o ..\build\files\sector_3.o
IF ERRORLEVEL 1 GOTO Error

ECHO.
ECHO Assembling loader
%osdk%\bin\xa -DASSEMBLER=XA loader.asm -o ..\build\files\loader.o
IF ERRORLEVEL 1 GOTO Error

::IF NOT EXIST BUILD\symbols GOTO NoSymbol

popd

:: Assemble the data
pushd resdata
ECHO.
ECHO Assembling resource data..
::%osdk%\bin\xa -DASSEMBLER=XA resource_data.s -o ..\build\files\resource_data.o

for %%f in (*.s) do (
ECHO Assembling %%f..
%osdk%\bin\xa -DASSEMBLER=XA %%f -o ..\build\files\%%~nf.o
IF ERRORLEVEL 1 GOTO Error
)

popd

:: Let's assemble the engine
echo Assembling OASIS...

::
:: Set the build paremeters
::
CALL osdk_config.bat

::
:: Launch the compilation of files
::
CALL %OSDK%\bin\make.bat %OSDKFILE%

pushd floppycode
:: Call FloppyBuilder once to create loader.cod
%osdk%\bin\FloppyBuilder build floppybuilderscript.txt

if "%FLOPPYPASS%"=="--" goto EndLoop
set FLOPPYPASS=%FLOPPYPASS%-
goto Loop

:EndLoop

All the times I end up with a header file containing the correct sector information, but with the actual data in disk containing different data (always the same I posted). Probably it is related to the thing that the compressed size is too close to the limit to include one extra sector or something like that.

My case is a bit special as I also include the header file with the information to get the uncompressed file size, so this is included in the loader and also in the main program. Maybe this will change as soon as I migrate to the new version of FloppyBuilder. If you have some suggestions to ease this transition... I am a bit lost here.

I don't know what else to do. If you have some ideas... else I will continue with the development with the extra bytes added or maybe with no compression and check later if the error persists.

[Dbug]

I knew the code is good enough as to detect if compressing was worth it (thought I'd say maybe check not only if the size is less but maybe have some kind of margin, such as if the compressed data uses less sectors).

True, if modulo 256 that does not change anything, I guess it's worth keeping it uncompressed.

All the times I end up with a header file containing the correct sector information, but with the actual data in disk containing different data (always the same I posted). Probably it is related to the thing that the compressed size is too close to the limit to include one extra sector or something like that.

What I suggest is that you look at the code of the FloppyBuilder, and look at it with new eyes.
I've been looking this code so much that I may just be missing something obvious.

The function is bool Floppy::WriteFile(const char *fileName,bool removeHeaderIfPresent,const std::map<std::string,std::string>& metadata) in
http://miniserve.defence-force.org/svn/ ... Floppy.cpp

Basically I write stuff to the disk directly, but I store the information used later to generate the header file at the same place, so I'm not sure how it gets out of sync.

If you can send me your project, I can take a look using your actual data.
Also, can you tell me which version of FloppyBuilder you are using?

[Chema]

I am still using version 0.018.

First the important bit: I made it work! Now some explanations

It is something related to the multiple passes... As I told you I include the floppy_description.h in my own code, so I have a copy of the sector information and it is the same as in the loader. Both wrong. However, after the build process if I look the floppy_description.h file it contains the CORRECT information.

So if it is generated twice (once at each pass) it is as if the one used in the build is the first one. As I am not sure how the 2 pass thing works, I wonder if the second pass does not detect that files have changed due to this included file and they are added as they ended-up in the first pass or something like that.

I put a PAUSE command inside the batch file and checked what happened to the floppy_description.h file. There are three different files being generated. The first one is a first guess, the second one is the one with the wrong data that ends up in the program and the third one is generated in the final build (and thus, not used).

So I added one more pass to the build process and it worked this time. I am quite sure I did that before (even 4 passes) with no success, but alas, it is good news!!

I would suggest that you generate a copy or temporal file with the information about sector/track/size whatever and check it against the last calculated to see if no more passes are needed.

[Dbug]

I should have been clearer in the documentation: At each pass you HAVE TO use the header file generated at this pass, if you use any other version you copied over, it will most probably create weird bugs. That was the whole point of the multiple passes: The header file content changes the content of the compiled files, which in turn changes their size and/or "compressability", which in turn impacts both the floppy content and the generate header file, which impacts the content of the compiled files... until it does not anymore

The whole system assumes that the list of files does not change, that the input script stays the same, and that the generate files are used in the subsequent passes.

I would suggest that you generate a copy or temporal file with the information about sector/track/size whatever and check it against the last calculated to see if no more passes are needed.

Well, my idea was to load the existing file before saving the new one, and if identical output a different error code to indicate that we don't need any more passes.
Could use ERRORLEVEL for that I guess.

[Dbug]

I am still using version 0.018

Well, there's been quite some changes since then:

0.19 - 2016/01/10
- Improved some error message to make them more useful when a problem happens.
- Made it possible to use the system without having to delete the build folder if for some reason the size of a sector file got too large.

1.0 - 2016/01/25
- Added a 'FormatVersion' command to help handle the lack of backward compatibility
- Added a 'WriteLoader' command to simplify the handling of loader specific parameters (the loader cannot be compressed, should not be in the directories, etc...)
- Three new defines are automatically created: FLOPPY_LOADER_TRACK, FLOPPY_LOADER_SECTOR and FLOPPY_LOADER_ADDRESS. They are designed to be used by the boot sectors to help load the loader.
- Added a new set of macro variables: {FileTrack}, {FileSector}, {FileSize} and {FileSizeCompressed}
- It is now possible to use the -D switch on the command line parameters to add a number of defines to the list of defines exported to the header file.

At the very least, update to 0.19, it's still the same format.

[Chema]

Mmmm I think I didn't explain myself correctly. I am not doing anything strange with the files. It was simply that three passes were needed, instead of just two

I have the intention to switch to version 1.0, Dbug. It's only that I need to find the courage to do so, because many things changed. That is why I was asking to a kind of quick start guide or list of things to change or look at

[Chema]

Dbug, maybe you overlooked my post in the Games forum about disk access... I should have posted it here, I think. (?)

Also if you could post or send me a version 1.0 of Floppy Builder I may start the migration. I know you attached one in this thread, but you said that version was not the last one (and still contained the uncompressed size in the tables).

[Chema]

Okay, I moved the discussion of FloppyBuilder here from the Games forum, to avoid polluting the Blake's 7 thread.

I made a minimalistic testbed for FloppyBuilder with a program that writes consecutive values to 7 pages in memory, then saves those pages to disk and loads them back. A second loop checks if the values are correct. If they are not, a red bar is displayed at the bottom of the screen. If they are, the program loops forever with all the screen in black

I removed compression and interleave and tried in Oricutron (and in Euphoric too). Everything worked perfectly. I attach the disk image. I moved to Cumulus (a clean empty image) and the red bar appeared. I also attach the disk image. If you examine both files with an hex editor you'll see differences in the gap length (the 0s and 22s) but the important thing is that several pages are wrong by 1 byte, starting in 1 instead of 0 (I think the first is ok). So 1 byte is missing from the stream and goes undetected.

Before you ask, the last byte is not again a 00, as if the stream continued with the following page, but a duplicated $ff. Then the next page starts again in 1, missing the first byte (a 0).

I also attach the sources. Dbug, this is the last version I am using. The code is not cleaned yet, sorry, but everything I changed is marked somehow with "Chema". It also includes my custom code for displaying the disk icon.

I did not test it with real hardware, but I bet it works.

The only differences with my old code here (check write_data) is that it was designed to write just one sector, so there was an external loop, it checked the termination of the WRITE_SECTOR command checking bit 7 in $314 as the FDC was configured to set IRQs, though the routines did not use it (OLDROUTINES is not defined). There is also a reminiscence of code which handled 512 byte sectors (Fabrice's code), which now I notice

I'd like to find a workaround for this... If someone can shed some light, I'd appreciate it. I have to concentrate on the game, or it won't be released in time!

[Dbug]

I removed compression and interleave and tried in Oricutron (and in Euphoric too). Everything worked perfectly. I attach the disk image. I moved to Cumulus (a clean empty image) and the red bar appeared. I also attach the disk image. If you examine both files with an hex editor you'll see differences in the gap length (the 0s and 22s) but the important thing is that several pages are wrong by 1 byte, starting in 1 instead of 0 (I think the first is ok). So 1 byte is missing from the stream and goes undetected.

What I suggest is first to try to fix the gaps to get a working floppy structure.

So based on what I've read on the other thread, the issue is that the code writes 0x22 instead of 0x4e, and that does not cause issues on real floppies because WriteDsk actually fixes that when it generates the floppy?

It should be relatively trivial to test.

FloppyBuilder.zip

for (int i=0;i<gap2-12;i++) trackbuf[offset++]=0x4E; //trackbuf[offset++]=0x22;

(104.12 KiB) Downloaded 246 times

It's a version identical to the 1.1, except the 0x22 replacement:

Code: Select all

for (int i=0;i<gap2-12;i++) trackbuf[offset++]=0x4E;   //trackbuf[offset++]=0x22;

Please try with this version, and tell me if that changes anything at all

If I did not understand what the problem was, then please explain me like if I was a 3 years old

[Symoon]

I second Dbug in trying to fix things one by one.
1st difference we noticed, let's try to check if it changes anything.

I tried to look anyway at the DSK files, as you explained the interleaves are very different one from the other, with a different amount of #22, a different amound of #00. But I also notice #E1#22 sometimes in Cumulus version, which is not in Oricutron version, in the sectors you updated (see it in the files around offset #1060). But it's there in all the other sectors apparently.
And it seems like it's when this #E122 is there that your sector data begins with #01 instead of #00.

Don't ask me why, just noticed that! All this seems quite messy for my little knowledge...