FloppyBuilder evolution

Anything related to the tools Tap2Wav, Tap2CD, Tap2Dsk, Sedoric Disc Manager, Tape Header Creator, WriteDsk, and generaly speaking tools related to the management of Oric data files and devices.
User avatar
Chema
Game master
Posts: 3013
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: FloppyBuilder evolution

Post by Chema »

Okay, I'll have a look at that new version as soon as possible and see what happens. Nice finding, Symoon... #E122 instead of what?

Dbug, I wish I knew what the problem is. Till now it seems that the differences in interleaving is a separate and maybe unrelated thing, but who knows. Maybe it is a combination of factors.
User avatar
Symoon
Archivist
Posts: 2301
Joined: Sat Jan 14, 2006 12:44 am
Location: Paris, France

Re: FloppyBuilder evolution

Post by Symoon »

Actually E122 seems to be the sector CRC; I don't know how to calculate it but this is my interpretation of what I see around offset #1060:
in Cumulus DSK, there are 2 bytes at #22 and 6 bytes at #00 missing in the "gaps" between sectors, so the sector datas begins 8 bytes earlier and end with a normal (I suppose!) CRC equal to #97FF, then follow what was the previously end of an empty sector: six bytes at zero, then the previous CRC, which is apparently #E122 when a sector data is filled by #00s.
So the new datas were shifted by 8 bytes earlier in the disk structure, and from there all the following sectors you record are messy with the 1st byte (#00) missing.

The question is: why are these #22 and #00 missing???
In Sedoric à Nu page 63, André says that the zone 22x #4E, 12x#00, 3x #A1 is strictly mandatory...

That doesn't strictly explain the missing 1st data byte at #00, but anyway there's a mess somewhere, either created by the fact that #22 replace #4E, either done by something else...
User avatar
Chema
Game master
Posts: 3013
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: FloppyBuilder evolution

Post by Chema »

Not sure about the $22 but the missing zeros are so because of the cumulus firmware: it erroneously uses 6 bytes to 0 instead of 12. But that is hardcoded so it should either always fail or always work.

It does not explain what we have here... at least not as far as I can see.
User avatar
Symoon
Archivist
Posts: 2301
Joined: Sat Jan 14, 2006 12:44 am
Location: Paris, France

Re: FloppyBuilder evolution

Post by Symoon »

Ouch, this Cumulus #00 bug is dirty :shock:

I see no other difference than #22/#4E (and the two extra #22) with what would be a Sedoric disk structure.

Here's how I see things: the problem can be in your test program (sorry ;) ), on FloppyBuilder with the identified #22/#4E difference giving unexpected results, elsewhere in FloppyBuilder, in Oricutron (could work by "chance" with specific programming), in Cumulus (giving the faulty result), in *your* Cumulus, or anywhere else.

To try and find the problem location, we need to reduce the amount of uncertain variables.
So in my view, if not already done, you can:
- replace Oricutron by Euphoric and compare if identical => will rise the probability that Oricutron is not "lucky"
- replace those #22 by #4E => 1st thing I'd try, this goes against all docs and can have a complete unkonwn effect. IMO you can't test and reach any certain conclusion if you know you're not using a standard structure.
- replace a FloppyBuilder disk by a Sedoric disk (same effect as above IMO)
- replace Cumulus by real floppy drives and compare if identical => the problem is not the Cumulus (but this test can be flawed by the way Writedisk works)
- put your test program somewhere for others to test on their hardware
-...

We have no explanation because it seems none of us knows how the complete chain works, but we see differences so we must act on the ones we can!
User avatar
Chema
Game master
Posts: 3013
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: FloppyBuilder evolution

Post by Chema »

Just tested this last version with the $4e. Same symptoms. Works in Oricutron, not with Cumulus. The sectors wrote by Cumulus are all but the first off by 1 byte (start in 1, instead of zero).

Didn't I tell that my code works on Eurphoric too? It does.

You can take any of the floppy images I posted and try yourself. If you see the flashing disk and the screen ends pitch black, it worked. If you see a red line at the bottom, it failed.

My code works in real floppies and with Jede's Telestrat and HxC, and also on the two emulators, so it should be something related to Cumulul... or MY cumulus...

I have been checking the firmware of Cumulus, and I cannot see the problem. It looks like either a bug when writing data to the SD file (unlikely, as it would fail in other cases too) or as if DRQ lines were active too long, so my code writes twice on the data register, but only at the beginning of the sector, which does not make sense. The signal is active for a fixed time, and it would most probably fail at random positions.

So again lost :(

EDIT: Added the last version, generated with Dbug's modification of FloppyBuilder using $4e
I am starting to think that the bug may be in the FAT driver, because after failing, resetting the Oric through Cumulus gives a "no system on disk" error, and I have to unplug the machine. The SD and FAT code is quite buggy, IIRC from when I played with it. Can any of you test with your Cumulus?
FBTEST-4e.dsk
(1000.25 KiB) Downloaded 415 times
User avatar
Symoon
Archivist
Posts: 2301
Joined: Sat Jan 14, 2006 12:44 am
Location: Paris, France

Re: FloppyBuilder evolution

Post by Symoon »

Chema wrote: Wed Aug 30, 2017 11:42 pmYou can take any of the floppy images I posted and try yourself. If you see the flashing disk and the screen ends pitch black, it worked. If you see a red line at the bottom, it failed.
Ah, thanks. I tried running a DSK file with Euphoric yesterday, saw the flashing loading disk but wasn't sure that something happened.
My code works in real floppies and with Jede's Telestrat and HxC, and also on the two emulators, so it should be something related to Cumulul... or MY cumulus...
Thanks for having tried with 4Es; it leaves behind the biggest visible question! We can free our minds of this one.
Will try your new DSK it with my Cumulus this weekend and let you know (don't have any Cumulus or floppy drive here :( )
I have been checking the firmware of Cumulus, and I cannot see the problem. It looks like either a bug when writing data to the SD file (unlikely, as it would fail in other cases too) or as if DRQ lines were active too long, so my code writes twice on the data register, but only at the beginning of the sector, which does not make sense. The signal is active for a fixed time, and it would most probably fail at random positions.
For the DSK I looked yesterday, it felt like something wrong happens in the disk structure (FAT or interleave counter, or whatever) after having written the 1st sector, because the 1st one has all the 256 bytes, and all the following ones begin with #01 instead of #00.

EDIT: before testing with my Cumulus, I'll rebuild a standard structure over your 4E test file (Cumulus modified it). I'm a bit obstinate but I don't want to leave any room for questions like "and if..." ;)
User avatar
Chema
Game master
Posts: 3013
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: FloppyBuilder evolution

Post by Chema »

Symoon wrote: Thu Aug 31, 2017 5:55 am [For the DSK I looked yesterday, it felt like something wrong happens in the disk structure (FAT or interleave counter, or whatever) after having written the 1st sector, because the 1st one has all the 256 bytes, and all the following ones begin with #01 instead of #00.
You may have a point here... My old routines were only sector based write_sector(address, sectornumber), basically, so I had to call them the number of times I needed to save things (a quick asm loop).

This new routine writes a number of sectors consecutively. It is basically the same, but the loop is inside the routine itself. I am doing basically the same operations, but maybe there are some delays (or lack of) causing timing problems? I will check anyway.

If anybody can do a quick test with his Cumulus, just to discard it is my unit only (I played with the firmware in the past, who knows...), I'd appreciate it.
User avatar
Symoon
Archivist
Posts: 2301
Joined: Sat Jan 14, 2006 12:44 am
Location: Paris, France

Re: FloppyBuilder evolution

Post by Symoon »

Chema wrote: Thu Aug 31, 2017 1:53 pmIf anybody can do a quick test with his Cumulus, just to discard it is my unit only (I played with the firmware in the past, who knows...), I'd appreciate it.
I'm travelling tomorrow and should be able to test tomorrow night or Sunday morning.
User avatar
Chema
Game master
Posts: 3013
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: FloppyBuilder evolution

Post by Chema »

Okay, spent a few minutes today with this and found something else that might help (or not!).

I checked the CRCs of the sectors and things are interesting. First of all, CRCs created by FloppyBuilder don't seem correct. They are all the same value, despite the sector contents. I checked the sources of FloppyBuilder and I think the CRC is computed for an empty sector and never again updated when writing data.

As writedsk would simply get the data and store it, without looking at the CRCs in the dsk file, it works. As most code simply ignore the CRCs, no problem in Oricutron or Euphoric (or even Cumulus, which I think does not check them either unless WD179X_CHECK_CRC is defined, which is not!).

Second, when writing Oricutron places the correct CRCs (I think) and those are equal to the ones generated by Cumulus (that also calculates them, see later) if the sectors written are the same, and different for the sectors that are 1 byte off! They are consistent, however, throughout all the file.

Now, Cumulus calculate the CRCs as he reads bytes sent by the Oric. Here is the code:

Code: Select all

			// Generate DRQ.
			generate_data_request();

			...
			
			/* TODO: Check if DRQ's have been serviced */
			// Buffer the data from the CPU and calculate CRC.
			pos = sector_buffer;
			cnt = length;
			crc = 0xE295;
			while (cnt)	
			{
				*pos = byte = read_data_register();
				crc = (crc << 8) ^ crc_table[(crc >> 8) ^ byte];
				generate_data_request();
				wait_for_next_byte(); 
				cnt --;
				pos ++;
			}
I removed the code that jumps over 22 bytes on the open file to make it clearer. Well, I cannot tell if the CRC is correct, but it is different from the one which corresponds to the full sector, so I *think* it is correct for a sector beginning with $01 and ending with two $ff, instead.

This would mean that the problem is not in the FAT driver or the way data is written, but on how it is read from the Oric (wait_for_next_byte makes a pause, read_data_register does exactly that - and yes: I have tried to add code to make sure DRQ goes to zero before checking it high again and write new data from the Oric's side and still fails).

Next thing I am going to try is a desperate version with some extra waiting between writing sectors. Just in case it is a problem with the signals, the bus, interference or any other thing like that.
User avatar
Symoon
Archivist
Posts: 2301
Joined: Sat Jan 14, 2006 12:44 am
Location: Paris, France

Re: FloppyBuilder evolution

Post by Symoon »

Forgive me in advance if the question is stupid, because I feel faaaaar away from what you're explaining here (never had a look at any code for Cumulus and my C language has become weak)
Chema wrote: Fri Sep 01, 2017 6:10 pm

Code: Select all

			while (cnt)	
			{
				(...)
				wait_for_next_byte(); 
				cnt --;
				(...)
			}
The fact that the system waits for the next byte BEFORE decreasing cnt, which stops the loop... Wouldn't that skip this byte? (and if this byte is the next #00...)
This seems possible regarding our problem:
- 1st read 100% ok => 1st write 100% ok,
- but then reads a last byte before stopping the writing (a #00)
- then loops starting reading at #01

(Mmmmh, maybe I'm just repeating the problem you were explaining and looking for...)
User avatar
Dbug
Site Admin
Posts: 4437
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Re: FloppyBuilder evolution

Post by Dbug »

Chema wrote: Fri Sep 01, 2017 6:10 pmI checked the CRCs of the sectors and things are interesting. First of all, CRCs created by FloppyBuilder don't seem correct. They are all the same value, despite the sector contents. I checked the sources of FloppyBuilder and I think the CRC is computed for an empty sector and never again updated when writing data.
You are correct, I totally forgot about it... mostly because none of the software actually cares about it, so I never noticed that :)

Definitely something that needs fixing, but I doubt it's the cause of your problems... if it was, it would not even load the game at all.

It's definitely possible that the issue is the writing code in Cumulus, but if that's the case it should be possible to test that with a normal Sedoric disk, like saving a HIRES picture, copying a file on another name, etc... if there's a problem we should have corruption of the file system, or other nasty stuff (missing bytes in the picture, ...)
User avatar
Chema
Game master
Posts: 3013
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: FloppyBuilder evolution

Post by Chema »

Symoon... you might have pinpointed the issue!!!

I never checked the loop (which is a bit of an abomination :lol: ) because: a/ the cnt var holds the number of bytes per sector, but it is not zero based, so the count is correct, b/my loop only writes 256 bytes then exits and a new SEEK command for the next sector is issued, ignoring what may happen there.

But you are right and the loop produces an extra DRQ signal. Usually this signal would be ignored, it is just a few cycles (as opposed to the pause between issuing it and reading the data register, which is -in Cumulus- a delay of 950us if I understood correctly).

If you issue a SEEK command after this (which the old code did and I bet most of the code based on writing sectors), this extra DRQ would definitely be ignored and no harm.

But floppybuilder code is intelligent and if track has not changed, it won't issue a SEEK command and simply issues a WRITE command and start watching for a DRQ. If this goes quickly enough as to get the extra DRQ, it will write the data register with the first byte when Cumulus is not yet reading data, and it will be missed (the following DRQ will make the Oric to write the second byte).

Oh, but all that makes no sense because my sector writing loop finishes checking the status bit and won't exit until it goes low, which would indicate the write command has finished on the Cumulus part... so it is not possible that it gets the extra DRQ... unless...

I checked the code on Cumulus and I see no trace of the busy flag being used at all! It correctly uses the IRQs but there is even a clear_status_flag which is commented out and the flag is not set to busy in Type II commands, so maybe we have something here.

Does it make sense?

At least it seems to explain things... It is possible to simply touch the loader in FloppyBuilder so it issues a SEEK command even when the track is correct (which is what is done in my old code, as this is not checked), and see if it solves the problems!!!

I don't know if I would be able to do it today. If not, I'll try to make a test tomorrow!!

In any case this code in cumulus is strange and maybe an update to the firmware correcting a few things (at least the number of zeros and the extra DRQ).
User avatar
Symoon
Archivist
Posts: 2301
Joined: Sat Jan 14, 2006 12:44 am
Location: Paris, France

Re: FloppyBuilder evolution

Post by Symoon »

Chema wrote: Sat Sep 02, 2017 8:31 am Symoon... you might have pinpointed the issue!!!
Cool if it can lead to something! That's luck, I got the idea because I was fighting with my progress bar code and some similar issues (decrease first then stop and loop, or decrease after, and so on ;) )
I don't know if I would be able to do it today. If not, I'll try to make a test tomorrow!!
Do you still need me to test the "4E DSK" with my Cumulus?
User avatar
Chema
Game master
Posts: 3013
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: FloppyBuilder evolution

Post by Chema »

Guys, I made a quick check, commenting the skip of the SEEK command in the loader of FloppyBuilder:

Code: Select all

	...	
	lda current_track                        ; Check if the drive is on the correct track		
	/************************ THIS ONE *********************/
	;PROTECT(FDC_track_register)
__fdc_track_1	
	;cmp FDC_track_register
	;beq stay_on_the_track
	/************************ OUT *********************/
retryseek
	PROTECT(FDC_data)
__fdc_data_1	
	sta FDC_data                             ; Set the new track		
	lda #CMD_Seek
	PROTECT(FDC_command_register)
__fdc_command_1
	sta FDC_command_register	
	jsr WaitCompletion			; CHEMA: Wait for the completion of the command
	...	
And guess what? IT WORKED!!!! It is clearly slower loading and saving (of course) so I should at least include a way so it is only skipped when saving or something, but I think the hypothesis was in the good direction!

@Symoon give it a try, but no need for you to modify the disk image. Just check what happens to discard it is only a timing issue in my unit.
User avatar
Dbug
Site Admin
Posts: 4437
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Re: FloppyBuilder evolution

Post by Dbug »

Nice find.
So technically... it's a Cumulus firmware bug?
Post Reply