Clock Signal — an Oric emulator for macOS and Linux

Comments, problems, suggestions about Oric emulators (Euphoric, Mess, Amoric, etc...) it's the right place to ask. And don't hesitate to give your tips and tricks that help using these emulations in the best possible way on your favorite operating system.
User avatar
NekoNoNiaow
Pilot Officer
Posts: 119
Joined: Sun Jan 15, 2006 10:08 pm
Location: Montreal, Canadia

Re: Clock Signal — an Oric emulator for macOS and Linux

Post by NekoNoNiaow » Fri May 25, 2018 8:10 am

iss wrote:
Fri May 25, 2018 7:45 am
BTW, WOZ format is good thing too, I've downloaded tons of images but I didn't find an actual converter DSK->WOZ, do you have such one? It would be perfect to have sources so I can compile and use it under Linux/Windows/macOS.
I am intrigued, why do you want to convert from DSK to WOZ if the emulator can read both?
If I remember correctly, the WOZ format is designed to reproduce/emulate the physical structure of the disk which other formats lack (even KryoFlux's).
So you would not gain anything by converting a DSK to WOZ, if that is even possible at all.

Ideally, you would want to generate WOZ files from original media to ensure they are properly reproducible, disk protection included.

Please correct me if I am wrong. ;)

User avatar
iss
Squad Leader
Posts: 669
Joined: Sat Apr 03, 2010 5:43 pm
Location: Bulgaria
Contact:

Re: Clock Signal — an Oric emulator for macOS and Linux

Post by iss » Fri May 25, 2018 9:31 am

NekoNoNiaow wrote:
Fri May 25, 2018 8:10 am
I am intrigued, why do you want to convert from DSK to WOZ if the emulator can read both?
The case is simple: I have already DSK image with some DOS3.3/8D-DOS software and this software needs to write back to the same image something in non-standard format (for instance a nibbelized bit stream but with custom address and data markers, or with custom number of sectors), so I need image format which supports data on "nibble level" and this is WOZ - if I have converter DSK->WOZ it will be easier to work with only one format. That's it ;).
PS: The case is not related to protection schemes and things like that...

ThomH
Flying Officer
Posts: 146
Joined: Thu Oct 13, 2016 9:55 pm

Re: Clock Signal — an Oric emulator for macOS and Linux

Post by ThomH » Fri May 25, 2018 2:45 pm

I can cobble together a DSK to WOZ converter from what's already in the emulator, though it might make sense to take a moment to simplify it a little. Apologies for NIB still being read-only, by the way; when I became aware of the Microdisc regression I just tied up some loose ends and released, and it just so happened that I had not yet worked on NIBs.

Re: the current processing cost of the emulator, it's gone up a little in this release when using 8DOS because I spotted that the Disk II state machine captures the contents of the data bus whenever it reaches the appropriate state, regardless of whether the processor actually writes to it. So in this second draft of 8DOS support, whenever the Disk II is active it is made aware of all data bus activity.

Caveats are that I'm perfectly happy spotting special cases to reduce the processing cost so these things are already implemented:
  • if the Disk II is in read mode but neither drive is enabled and the shift register has filled with 0s then the component is considered 'asleep' and no longer actively clocked;
  • if the Disk II is in sense write protect mode and the shift register has already filled with either 1s or 0s as the write protect status indicates, the Disk II is similarly marked as asleep.
Which means, sure, I'm using the original state machine ROM for the actual activity but I'm also making a priori assertions about the contents of that ROM. If the assertions are correct then the only difference this would ever make to a machine is if an author wanted to test a replacement state machine ROM.

And the optimisation that's coming will be to introduce a third state between asleep and awake of needing just-in-time execution only. So that in read mode, which never samples the data bus, only loads it upon request, the Disk II isn't just always in the loop for bus activity, and rather than being consistently clocked goes back to being clocked just-in-time. I.e. if you access the Disk II on cycle X then again on cycle X+M, then the Disk II does absolutely nothing for the interim, then performs M cycles all at once before the next access. There's zero observable difference to software in the emulated machine but it's a lot easier on your host machine's instruction pipeline and cache.

EDIT: as to emulated loading speed, I see that I've failed to account for the difference in Oric and Apple II clock speeds, which makes the disks spin about 2.2% too slowly. So mine are going at about 293 RPM. That's an easy fix. Otherwise it should be the same. Unless there's a tendency for Disk IIs to be a little faster than 300RPM?

User avatar
NekoNoNiaow
Pilot Officer
Posts: 119
Joined: Sun Jan 15, 2006 10:08 pm
Location: Montreal, Canadia

Re: Clock Signal — an Oric emulator for macOS and Linux

Post by NekoNoNiaow » Sun May 27, 2018 12:35 am

iss wrote:
Fri May 25, 2018 9:31 am
The case is simple: I have already DSK image with some DOS3.3/8D-DOS software and this software needs to write back to the same image something in non-standard format (for instance a nibbelized bit stream but with custom address and data markers, or with custom number of sectors), so I need image format which supports data on "nibble level" and this is WOZ - if I have converter DSK->WOZ it will be easier to work with only one format. That's it ;).
Thanks for the explanation, indeed if you want to write back to the disk in a special way, the format needs to support it so this makes perfect sense.

ThomH
Flying Officer
Posts: 146
Joined: Thu Oct 13, 2016 9:55 pm

Re: Clock Signal — an Oric emulator for macOS and Linux

Post by ThomH » Sun May 27, 2018 5:11 am

NekoNoNiaow wrote:
Sun May 27, 2018 12:35 am
iss wrote:
Fri May 25, 2018 9:31 am
The case is simple: I have already DSK image with some DOS3.3/8D-DOS software and this software needs to write back to the same image something in non-standard format (for instance a nibbelized bit stream but with custom address and data markers, or with custom number of sectors), so I need image format which supports data on "nibble level" and this is WOZ - if I have converter DSK->WOZ it will be easier to work with only one format. That's it ;).
Thanks for the explanation, indeed if you want to write back to the disk in a special way, the format needs to support it so this makes perfect sense.
Potentially helpful in the interim: what you can write to disk in the emulator is completely disjoint from what it can store from that disk into a disk image. You'll just lose whatever the image isn't expressive enough for in the transcription. So you can load a DSK, write whatever you want, and it'll be retained in full fidelity in memory for as long as the emulator is running. That might provide a limited testing window?

ThomH
Flying Officer
Posts: 146
Joined: Thu Oct 13, 2016 9:55 pm

Re: Clock Signal — an Oric emulator for macOS and Linux

Post by ThomH » Wed Jun 06, 2018 1:22 am

iss wrote:
Fri May 25, 2018 9:31 am
NekoNoNiaow wrote:
Fri May 25, 2018 8:10 am
I am intrigued, why do you want to convert from DSK to WOZ if the emulator can read both?
The case is simple: I have already DSK image with some DOS3.3/8D-DOS software and this software needs to write back to the same image something in non-standard format (for instance a nibbelized bit stream but with custom address and data markers, or with custom number of sectors), so I need image format which supports data on "nibble level" and this is WOZ - if I have converter DSK->WOZ it will be easier to work with only one format. That's it ;).
PS: The case is not related to protection schemes and things like that...
I was distracted by other issues arising but give https://github.com/tomharte/dsk2woz a go.

User avatar
NekoNoNiaow
Pilot Officer
Posts: 119
Joined: Sun Jan 15, 2006 10:08 pm
Location: Montreal, Canadia

Re: Clock Signal — an Oric emulator for macOS and Linux

Post by NekoNoNiaow » Wed Jun 06, 2018 5:52 am

ThomH wrote:
Sun May 27, 2018 5:11 am
Potentially helpful in the interim: what you can write to disk in the emulator is completely disjoint from what it can store from that disk into a disk image. You'll just lose whatever the image isn't expressive enough for in the transcription. So you can load a DSK, write whatever you want, and it'll be retained in full fidelity in memory for as long as the emulator is running. That might provide a limited testing window?
Do you plan to eventually implement save states? I have not found a way to generate them so I assume the feature is not implemented.

If you do, they would offer a way to have persistent disks without having to use a particular disk format. ;)
The issue being that one would need to maintain a different save state for each disk written to.

Which leads me to the logical followup question:
Assuming save states are implemented and they retain all the necessary data pertaining to the disk present in the emulated drive,
if the user ejects a disk and replaces it with another, will the emulator somehow keep the memory of the ejected disk so that it can be reinserted?

On a physical machine, obviously this is a basic expectation as disks do not vanish into thin air once ejected.
But on an emulator which does not have a dedicated disk format, they would vanish.

So, the tortuous conclusion to which I want to arrive is that an accurate emulator with save states has no choice but to implement a disk save format compatible with its internal disk model, at least if it wants to offer the same experience as the physical machine.

I am sure there are shorter paths to this conclusion but I like that one. ;)

ThomH
Flying Officer
Posts: 146
Joined: Thu Oct 13, 2016 9:55 pm

Re: Clock Signal — an Oric emulator for macOS and Linux

Post by ThomH » Wed Jun 06, 2018 2:14 pm

No, I've never implemented saved states; sometimes I oscillate between popping up a warning to the user that they should let me resave their work as WOZ/FDI/as relevant in order best to preserve their changes, but actually none of those quite fully captures exactly everything — e.g. both assume a consistent data rate across each track, which almost never happens if you've written data as the original disk image almost never came from a machine with an exactly-300RPM drive, whereas the emulator writes with mathematical precision. Also there are some formats that are sector dumps but sometimes offer multiple samples of a sector as a way of capturing weak/fuzzy bits, which is a whole extra dimension of storage requirement.

It is definitely weird if what's on your disk mutates when you eject it from the drive, but it's also weird if it does so as soon as you send it to somebody else so I almost certainly need to actually say something. And if I'm saying something then I guess I can offer an option, even though I hate options.

As to the immediate plans for development though, I'm currently most concerned about the deprecation of OpenGL in macOS 10.14. That means adding an extra layer of separation into the CRT as a foundational assumption had been that OpenGL is the cross-platform abstraction. Then the OpenGL stays for SDL, at least unless and until something smarter presents itself as an option there, and I'll need to learn some Metal for macOS. I had a quick glance and it looks like a reasonable API; ditto for Vulkan. Also I suppose it'll finally give me the hooks I need to write an ES 2.0 back-end for hypothetical Raspberry Pi deployment. If I'm not just too slow all around.

P.s. a slight change to dsk2woz is scheduled for tonight: at the minute it writes very thin tracks. Approximately as if you'd written out a DSK with an 80-track drive. Sort of. So I need to fatten them. It's a trivial change — WOZ uses a level of indirection to identify the data that falls beneath each 0.25 of a normal complete-track stepping size, so it's just changing what I write into that table.

EDIT: scratch that; a pull request came in to make that change for me. So it's done already.

User avatar
iss
Squad Leader
Posts: 669
Joined: Sat Apr 03, 2010 5:43 pm
Location: Bulgaria
Contact:

Re: Clock Signal — an Oric emulator for macOS and Linux

Post by iss » Wed Jun 06, 2018 7:48 pm

Thanks for the small but very nice tool dsk2woz! I saw your announce on comp.sys.apple2 too :).
It works well but ... back to Clock Signal - is currently Disk][ read-only or something with the write-protection switch is wrong?
clk-writeprotect.jpg
Attached archive contains DSK, NIB and WOZ images for Oric + Disk][ controller.
This is a calibration tool for drive rotation speed - originally it's for Apple but I disassembled it and the port for Oric was fun and easy. I plan to use this program for calibration of real drives but ... I realized that I need first to calibrate the program itself too - with 2 drives (which I know are really good) - the program reads ~196,4 ms (instead of 200.0 ms) and I'm wondering is this difference because of the different CPU clock? :).
The program writes custom data over the track 0, so you'll need backup if you want to run it twice ;).
And this is the reason to ask you for WOZ converter.

I will check the real drive speed with oscilloscope and with frequency counter connected to the index hole sensor but this will happen during the weekend.

BTW, I can't open NIB's - the emulator fails with error message: terminate called after throwing an instance of 'std::bad_alloc' immediately when loading the NIB file.
Attachments
zugmd.zip
(15.78 KiB) Downloaded 20 times

ThomH
Flying Officer
Posts: 146
Joined: Thu Oct 13, 2016 9:55 pm

Re: Clock Signal — an Oric emulator for macOS and Linux

Post by ThomH » Wed Jun 06, 2018 8:03 pm

iss wrote:
Wed Jun 06, 2018 7:48 pm
Thanks for the small but very nice tool dsk2woz! I saw your announce on comp.sys.apple2 too :).
It works well but ... back to Clock Signal - is currently Disk][ read-only or something with the write-protection switch is wrong?
It was initially but that is supposed to be fixed. Well, partially fixed, there's a bug in the WOZ support if you write to a track that does not already have contents. I'll use the attached images to explore.
iss wrote:
Wed Jun 06, 2018 7:48 pm
This is a calibration tool for drive rotation speed - originally it's for Apple but I disassembled it and the port for Oric was fun and easy. I plan to use this program for calibration of real drives but ... I realized that I need first to calibrate the program itself too - with 2 drives (which I know are really good) - the program reads ~196,4 ms (instead of 200.0 ms) and I'm wondering is this difference because of the different CPU clock? :).
Yes, but it's (hopefully) fixed in the code. The emulated drives take a discrete clock from the machine that owns them, which means you have to tell them their clock rate, and I'd failed to supply a different one for the Oric than the Apple II.

Almost certainly not relevant to a rotation speed test, but potentially to be wary of for other types of speed test: if you're starting with a DSK and commuting it to a WOZ with my converter is that after writing all the sectors it then just pads the track with sync words until it has at least 50,000 bits. So without having checked, it might end up with anywhere between 50,000 and 50,009. Then, of course, the emulator will obligingly size the bits to whatever size they need to be to fill the track.
iss wrote:
Wed Jun 06, 2018 7:48 pm
BTW, I can't open NIB's - the emulator fails with error message: terminate called after throwing an instance of 'std::bad_alloc' immediately when loading the NIB file.
At one point they were working but my animosity towards them has probably manifested as negligent testing. Will try harder.

ThomH
Flying Officer
Posts: 146
Joined: Thu Oct 13, 2016 9:55 pm

Re: Clock Signal — an Oric emulator for macOS and Linux

Post by ThomH » Thu Jun 07, 2018 4:01 pm

Time is very tight this week, but things worth saying immediately:
  • in a case of accidental fuzz testing, I discovered that the WOZs produced by dsk2woz were truncating track bit count to produce the byte count (WOZ stores both) rather than rounding up. Which caused my emulator to perform an out-of-bounds read. Fixed at both ends.
  • I made some progress with NIBs, but still hate them.
The specific issue with NIBs: you are given 6656 bytes of data to represent each track. Those bytes represent all bytes read from the disk while in perfect stream synchronisation. They omit the synchronisation bits.

So you're starting with 53,248 bits of information. You definitely have to add more, because the synchronisation bits have to go somewhere.

If you were not to assume anything about the format of the track then any and all words might be followed by one or two sync bits. But if you put only a single sync bit after every single byte then you're now trying to cram 59,904 bits onto a single track. You've exceeded both the actual media density and what can be handled by the Disk II's PLL at 300RPM.

Suppose you make a limited data-format assumption and say that only patches of five-of-more FFs are sync words. Then you still frequently end up with more bits than can be parsed. Especially if any sector contents are ever padded with FFs or contain a lookup table with many FFs.

Okay, so you've got to make at least one more assumption. You can assume the Apple sector and header prologues, and that only FFs immediately before those are sync words. Even then you can't afford to treat all the FFs as sync words on most disk images before you blow your track bit budget.

So if you're me you then arrive at assuming you know the prologues, and marking at most five FFs before a prologue for extra sync bits. For Apple 6-and-2 encoding that's usually going to be only 2*5*16 = 160 extra vits per track, for 53,408 bits total. Which is pushing it but within bounds.

But then you're at NIB support only for disks with the conventional 6-and-2 encoding, including an assumption of the ordinary sector and header patterns. So what did you gain over DSK? For all that, nothing that wouldn't more intelligently have been stored by just including headers with sectors in a DSK-esque format. Like, say, the Amstrad CPC .DSK.

But classic emulators, that aren't actually all that interested in how the hardware works, just spool a new byte from the file every time the CPU reads the data register. Magic alignment! But the only way I can think of to reproduce that is a hardware hack: use the 59,904 that give every single byte two sync bits, slow down disk rotation (to approximately 280RPM, I think, but don't quote that; it's based on the Disk II state machine offering plus or minus an eighth on flux window lengths, which I feel I learnt but don't have a source to hand), and just write protect the whole thing because obviously writing real data on top would make the extended track length very odd.

I saw no issues opening NIBs as was; the contents aren't even loaded into memory, they're read from disk on demand, and I ran with both the Clang address sanitiser and the undefined behaviour analyser attached. But I'll keep at it.

ThomH
Flying Officer
Posts: 146
Joined: Thu Oct 13, 2016 9:55 pm

Re: Clock Signal — an Oric emulator for macOS and Linux

Post by ThomH » Fri Jun 29, 2018 7:20 pm

Update on this: in the latest release I've found that the rotation test disks trip up the internal precision for modified disk tracks, leading the emulator to perform incorrectly.

I've decided it's because I'm modelling them poorly. Hindsight shall be applied — I'm going to switch representations.

The technical stuff: ordinary tracks are one or more PCM-sampled segments. It'll be more than one if the track contains blocks of data sampled at different rates — Commodore machines are a practical example where this might be applicable. For Disk II disks there's always going to be only one segment.

Modified tracks are stored as (i) the underlying original; plus (ii) a list of all writes. Writes are stored as start position + new segment, and the collection is flattened so that it's a non-overlapping list of time periods that reference segments.

The nature of the latest precision error is somewhere in the logic for realising that the end of a patch has been reached. All times are stored as quotients but even if they were regular floats or doubles I'm not sure I'd be doing too much better.

So I've decided that the real issue here is: I've overcomplicated the task. All that maintaining and navigating at an extra level of indirection is just unnecessary, especially it's unnecessarily hard to test and visually audit. In any case, real disks are not exact, so I needn't be. If and when a machine starts writing to the disk I'll just map that track to a very-high precision representation, and directly manipulate that. Like 200,000 or 400,000 or something like that samples per track. Then let data of different rates just be aliased in that.

Nothing consuming data from tracks has any concept of the storage underneath so I'm hoping it won't be too difficult. Even the file format handlers don't make any assumptions. E.g. both WOZ and HFE output are generated by running a PLL over the track to rationalise it down to data rates those files tend to deal in and to try to be sensible about different data rates. Overwriting data of one rate with data of another is an unavoidable issue when simulating real hardware with many of these formats, as they provide a captured data density but the density at which machines write is fixed.

So then I'll see whether drive rates start to be reported properly by the disk images above. If so I'll be happy and publish a new release.

User avatar
Chema
Game master
Posts: 2282
Joined: Tue Jan 17, 2006 10:55 am
Location: Gijón, SPAIN
Contact:

Re: Clock Signal — an Oric emulator for macOS and Linux

Post by Chema » Fri Jun 29, 2018 10:51 pm

This has gone too technical for me :) I am impressed by your work and still wonder if there will be a Windows /Linux version :)

User avatar
Dbug
Site Admin
Posts: 2701
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Re: Clock Signal — an Oric emulator for macOS and Linux

Post by Dbug » Sat Jun 30, 2018 9:15 am

I'm just a silent lurker, it's interesting to follow, but I don't really have anything to comment on :D

User avatar
NekoNoNiaow
Pilot Officer
Posts: 119
Joined: Sun Jan 15, 2006 10:08 pm
Location: Montreal, Canadia

Re: Clock Signal — an Oric emulator for macOS and Linux

Post by NekoNoNiaow » Mon Jul 02, 2018 10:34 am

I also love reading these reflections even if I sometimes do so diagonally. ;)

I think your (ThomH) approach is the one I prefer when tackling emulation of old machines. Not only is it more accurate (which I favor) but it also ultimately allows for a mix of software and hardware constructions. With CLK, it is conceivable to run the emulator core in software while still allowing vintage peripherals to provide the input signals. Naive emulation approaches virtualize the hardware and as such cannot be easily interfaced with the actual hardware because they just have no notion of hardware signals.

A practical example of such an advantage is that it would be relatively easy to port CLK to the Raspberry PI to emulate an entire Oric with all of its expansion buses, thus allowing original hardware to be connected to it. Given the current popularity "mini/classic" devices (C64 mini, (S)NES classic, etc.) I think this is quite an important advantage.

To focus more on the topic of disk format though, it is interesting that despite years of experience with complex formats on floppy disks media (mainly on the Amiga where the controller allows bit precise control of what's read/written to the media) and of the problems that come with it, there are still formats which are not flexible enough to accommodate the capabilities of the actual hardware. It would seem that going with a format which allows encoding what is sent to the hardware and how it stores it (and conversely, also encoding how it will be read) would be the way to go rather than work at a logical level but yet we are still grappling with the same issues.

Ah, those lazy coders. ;)

Post Reply