FloppyBuilder evolution

Anything related to the tools Tap2Wav, Tap2CD, Tap2Dsk, Sedoric Disc Manager, Tape Header Creator, WriteDsk, and generaly speaking tools related to the management of Oric data files and devices.
User avatar
Chema
Game master
Posts: 1972
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: FloppyBuilder evolution

Post by Chema » Thu Jul 07, 2016 12:39 pm

Dbug, are you aware of any bug in the unpacking routine in the first version of FloppyBuilder? I am facing a strange bug which prevents things to load that appears and disappears as I enter/edit new code or add extra bytes. The bug disappears if I remove the FilePack option.

I know I have to move to the latest version, but I need time and energy to migrate.. Could you provide a quick and dirty guide about how to do it? :)

User avatar
Dbug
Site Admin
Posts: 2324
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Re: FloppyBuilder evolution

Post by Dbug » Thu Jul 07, 2016 11:06 pm

Not that I'm aware, but it's always possible :)

If you could write a minimal test case that exhibits the problem, would be helpful.

User avatar
Chema
Game master
Posts: 1972
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: FloppyBuilder evolution

Post by Chema » Fri Jul 08, 2016 12:13 pm

Dbug wrote:Not that I'm aware, but it's always possible :)

If you could write a minimal test case that exhibits the problem, would be helpful.
That will be difficult, Dbug, but I will try to see if I can narrow the issue a bit more. The thing is that I have to add extra bytes to the main program (in a certain range, no more no less :) ) so the loading of another part from disk works (part which has not changed). That is quite strange unless the loader code is disturbed somehow or the main program code has been decompressed with some kind of error.

Now that I am writing this... I could try to examine the first 16K of a dump both loaded with compression and without it... they should be the same....

User avatar
Chema
Game master
Posts: 1972
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: FloppyBuilder evolution

Post by Chema » Fri Jul 08, 2016 2:31 pm

Comparing dumps showed nothing strange. But I found something else.

The file which fails loading correctly is not compressed (it is small). It is number 2 in the data arrays, just after the main program which is either compressed or not depending on the floppy builder option. This main program is number 1 (number 0 is the loader).

When compressing and with no extra bytes added the generated floppy_description.h file the sector info is:
FileStartSector .byt 4,7,7,8,9,10,11,12,4,10,12,4,6,7,8,10,13,14,9,7,1,8,16,7,2,11

But the debugger shows that the stored information for the start sector is :

04 07 06 07 08 09 0A 0B

The thing is that if I add the extra bytes to the main program, the compressed size is (believe it or not) smaller (12800 vs 12804) enough to make the header file information now:
FileStartSector .byt 4,7,6,7,8,9,10,11,3,9,11,3,5,6,7,9,12,13,8,6,17,7,15,6,1,10

It is quite strange because I think this information ends up in the loader code with an #include directive, so either it is using a different (older?) header file or I have no idea what is going on. In addition I also include the same file for my directory (yeah, something that should be changed I know) and it is also wrong (in the debugger both memory areas show the same information).

I tried deleting the header file and the build folder before compiling, with no success.

I am quite sure that it is reading the end of the previous file, as it has some bytes of information and the rest are all zeros.

So if this all makes sense then it is a good hint towards the bug. Else it is elsewhere and I simply made some of the tests wrong.

P.S. As a side note if I remove compression the sector information is correct in the file and in the debugger, although it is (of course) different from the one above.

Edit 2: 12800/256=50 sectors exactly.

User avatar
Dbug
Site Admin
Posts: 2324
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Re: FloppyBuilder evolution

Post by Dbug » Fri Jul 08, 2016 7:11 pm

Hmm.

FloppyBuilder when asked to compress files tries to compress them, but if the compressed size is equal or largest to the non compressed one, then the file is stored uncompressed.

Code: Select all

  std::vector<unsigned char> compressedBuffer;
  if (m_CompressionMode==e_CompressionFilepack)
  {
    // So the user requested FilePack compression.
    // Great, we can do that.
    compressedBuffer.resize(fileSize);
    gLZ77_XorMask=0;
    size_t compressedFileSize=LZ77_Compress(fileData,compressedBuffer.data(),fileSize);
    if (compressedFileSize<fileSize)
    {
      // We actually did manage to compress the data
      fileData=compressedBuffer.data();
      fileSize=compressedFileSize;

      fileEntry.m_CompressionMode=e_CompressionFilepack;
      fileEntry.m_StoredFileSize=compressedFileSize;
      fileEntry.m_SectorCount=(fileSize+255)/256;
    }
  }
if it did not manage to compress then it uses the default parameters:

Code: Select all

  fileEntry.m_StoredFileSize=fileSize;
  fileEntry.m_FinalFileSize   =fileSize;
  fileEntry.m_SectorCount=(fileSize+255)/256;
  fileEntry.m_FilePath   =fileName;
  fileEntry.m_CompressionMode=e_CompressionNone;
The thing is that if I add the extra bytes to the main program, the compressed size is (believe it or not) smaller (12800 vs 12804)
Well, LZ77 is based on pattern matching and repetition, so adding bytes in some places may just may make this place match another one.
When compressing and with no extra bytes added the generated floppy_description.h file the sector info is:
FileStartSector .byt 4,7,7,8,9,10,11,12,4,10,12,4,6,7,8,10,13,14,9,7,1,8,16,7,2,11
But the debugger shows that the stored information for the start sector is :
04 07 06 07 08 09 0A 0B
Just one question: Are you using the FilePack as in my demos, or did you integrate it in your own build script? If you integrated in your build script, did you do the correct number of loops so all the programs that include FloppyBuilder generated files actually get recompiled/reassembled with the correct data?

Basically the reason for the multiple passes is that in order to be able to allocate sectors efficiently and without waste, the program needs to know the size of each file, and since some of the files include some FloppyBuilder information (which is dependent of the build script) it needs to build the whole project multiple time.

If you don't do it the right number of time, you get out of sync information.

What has to be rebuilt multiple time is only stuff that includes generated data, the rest is "binary stable" and will not change from one build to another :)

User avatar
Chema
Game master
Posts: 1972
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: FloppyBuilder evolution

Post by Chema » Fri Jul 08, 2016 11:21 pm

I knew the code is good enough as to detect if compressing was worth it (thought I'd say maybe check not only if the size is less but maybe have some kind of margin, such as if the compressed data uses less sectors).

It must be something related to the chicken-and-egg problem. I am using my own build batch file, but based on yours, so calling FloppyBuilder twice (I have tried making 3 and 4 iterations with no different results). Here is my code:

Code: Select all

:: Call FloppyBuilder once to create loader.cod
%osdk%\bin\FloppyBuilder init floppybuilderscript.txt


set FLOPPYPASS=-

:Loop
echo %FLOPPYPASS%
:: Call XA to rebuild the loader
ECHO.
ECHO Assembling bootsectors
%osdk%\bin\xa -DASSEMBLER=XA sector_1-jasmin.asm -o ..\build\files\sector_1-jasmin.o
IF ERRORLEVEL 1 GOTO Error
%osdk%\bin\xa -DASSEMBLER=XA sector_2-microdisc.asm -o ..\build\files\sector_2-microdisc.o
IF ERRORLEVEL 1 GOTO Error
%osdk%\bin\xa -DASSEMBLER=XA sector_3.asm -o ..\build\files\sector_3.o
IF ERRORLEVEL 1 GOTO Error

ECHO.
ECHO Assembling loader
%osdk%\bin\xa -DASSEMBLER=XA loader.asm -o ..\build\files\loader.o
IF ERRORLEVEL 1 GOTO Error

::IF NOT EXIST BUILD\symbols GOTO NoSymbol

popd

:: Assemble the data
pushd resdata
ECHO.
ECHO Assembling resource data..
::%osdk%\bin\xa -DASSEMBLER=XA resource_data.s -o ..\build\files\resource_data.o

for %%f in (*.s) do (
ECHO Assembling %%f..
%osdk%\bin\xa -DASSEMBLER=XA %%f -o ..\build\files\%%~nf.o
IF ERRORLEVEL 1 GOTO Error
)

popd

:: Let's assemble the engine
echo Assembling OASIS...

::
:: Set the build paremeters
::
CALL osdk_config.bat

::
:: Launch the compilation of files
::
CALL %OSDK%\bin\make.bat %OSDKFILE%

pushd floppycode
:: Call FloppyBuilder once to create loader.cod
%osdk%\bin\FloppyBuilder build floppybuilderscript.txt

if "%FLOPPYPASS%"=="--" goto EndLoop
set FLOPPYPASS=%FLOPPYPASS%-
goto Loop

:EndLoop
All the times I end up with a header file containing the correct sector information, but with the actual data in disk containing different data (always the same I posted). Probably it is related to the thing that the compressed size is too close to the limit to include one extra sector or something like that.

My case is a bit special as I also include the header file with the information to get the uncompressed file size, so this is included in the loader and also in the main program. Maybe this will change as soon as I migrate to the new version of FloppyBuilder. If you have some suggestions to ease this transition... I am a bit lost here.

I don't know what else to do. If you have some ideas... else I will continue with the development with the extra bytes added or maybe with no compression and check later if the error persists.

User avatar
Dbug
Site Admin
Posts: 2324
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Re: FloppyBuilder evolution

Post by Dbug » Sat Jul 09, 2016 11:09 am

Chema wrote:I knew the code is good enough as to detect if compressing was worth it (thought I'd say maybe check not only if the size is less but maybe have some kind of margin, such as if the compressed data uses less sectors).
True, if modulo 256 that does not change anything, I guess it's worth keeping it uncompressed.
Chema wrote:All the times I end up with a header file containing the correct sector information, but with the actual data in disk containing different data (always the same I posted). Probably it is related to the thing that the compressed size is too close to the limit to include one extra sector or something like that.
What I suggest is that you look at the code of the FloppyBuilder, and look at it with new eyes.
I've been looking this code so much that I may just be missing something obvious.

The function is bool Floppy::WriteFile(const char *fileName,bool removeHeaderIfPresent,const std::map<std::string,std::string>& metadata) in
http://miniserve.defence-force.org/svn/ ... Floppy.cpp

Basically I write stuff to the disk directly, but I store the information used later to generate the header file at the same place, so I'm not sure how it gets out of sync.

If you can send me your project, I can take a look using your actual data.
Also, can you tell me which version of FloppyBuilder you are using?

User avatar
Chema
Game master
Posts: 1972
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: FloppyBuilder evolution

Post by Chema » Sat Jul 09, 2016 8:29 pm

I am still using version 0.018.

First the important bit: I made it work! Now some explanations :)

It is something related to the multiple passes... As I told you I include the floppy_description.h in my own code, so I have a copy of the sector information and it is the same as in the loader. Both wrong. However, after the build process if I look the floppy_description.h file it contains the CORRECT information.

So if it is generated twice (once at each pass) it is as if the one used in the build is the first one. As I am not sure how the 2 pass thing works, I wonder if the second pass does not detect that files have changed due to this included file and they are added as they ended-up in the first pass or something like that.

I put a PAUSE command inside the batch file and checked what happened to the floppy_description.h file. There are three different files being generated. The first one is a first guess, the second one is the one with the wrong data that ends up in the program and the third one is generated in the final build (and thus, not used).

So I added one more pass to the build process and it worked this time. I am quite sure I did that before (even 4 passes) with no success, but alas, it is good news!! :D

I would suggest that you generate a copy or temporal file with the information about sector/track/size whatever and check it against the last calculated to see if no more passes are needed.

User avatar
Dbug
Site Admin
Posts: 2324
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Re: FloppyBuilder evolution

Post by Dbug » Sat Jul 09, 2016 9:37 pm

I should have been clearer in the documentation: At each pass you HAVE TO use the header file generated at this pass, if you use any other version you copied over, it will most probably create weird bugs. That was the whole point of the multiple passes: The header file content changes the content of the compiled files, which in turn changes their size and/or "compressability", which in turn impacts both the floppy content and the generate header file, which impacts the content of the compiled files... until it does not anymore :D

The whole system assumes that the list of files does not change, that the input script stays the same, and that the generate files are used in the subsequent passes.
I would suggest that you generate a copy or temporal file with the information about sector/track/size whatever and check it against the last calculated to see if no more passes are needed.
Well, my idea was to load the existing file before saving the new one, and if identical output a different error code to indicate that we don't need any more passes.
Could use ERRORLEVEL for that I guess.

User avatar
Dbug
Site Admin
Posts: 2324
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Re: FloppyBuilder evolution

Post by Dbug » Sat Jul 09, 2016 9:39 pm

Chema wrote:I am still using version 0.018
Well, there's been quite some changes since then:

0.19 - 2016/01/10
- Improved some error message to make them more useful when a problem happens.
- Made it possible to use the system without having to delete the build folder if for some reason the size of a sector file got too large.

1.0 - 2016/01/25
- Added a 'FormatVersion' command to help handle the lack of backward compatibility
- Added a 'WriteLoader' command to simplify the handling of loader specific parameters (the loader cannot be compressed, should not be in the directories, etc...)
- Three new defines are automatically created: FLOPPY_LOADER_TRACK, FLOPPY_LOADER_SECTOR and FLOPPY_LOADER_ADDRESS. They are designed to be used by the boot sectors to help load the loader.
- Added a new set of macro variables: {FileTrack}, {FileSector}, {FileSize} and {FileSizeCompressed}
- It is now possible to use the -D switch on the command line parameters to add a number of defines to the list of defines exported to the header file.

At the very least, update to 0.19, it's still the same format.

User avatar
Chema
Game master
Posts: 1972
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: FloppyBuilder evolution

Post by Chema » Sun Jul 10, 2016 11:52 am

Mmmm I think I didn't explain myself correctly. I am not doing anything strange with the files. It was simply that three passes were needed, instead of just two :)

I have the intention to switch to version 1.0, Dbug. It's only that I need to find the courage to do so, because many things changed. That is why I was asking to a kind of quick start guide or list of things to change or look at :)

User avatar
Chema
Game master
Posts: 1972
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: FloppyBuilder evolution

Post by Chema » Wed Sep 14, 2016 2:10 pm

Dbug, maybe you overlooked my post in the Games forum about disk access... I should have posted it here, I think. (?)

Also if you could post or send me a version 1.0 of Floppy Builder I may start the migration. I know you attached one in this thread, but you said that version was not the last one (and still contained the uncompressed size in the tables).

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests