Novalight - very fast tape loading

[Symoon]

Well, in fact what I have is the first Atmos ROM version, which does the right thing, and it is not bugged It checks for errors and stops loading if it finds them. What the Oric people did, after realizing that there were loading errors still, was to remove the error checking altogether in the second version 1.1b!

That is what the small program does... sets the IRQ vector to point to a small routine that clears the error flag... good job guys!

Back to this subject.
Actually there are two modifications that were done on the "1.1b" ROM in tape routines:
1- stop checking parity errors during the synchro (bytes 16 16 16 ... 16 16 24)
2- only display "Errors Found" and don't stop the program execution

Chema, I now understand better why you say it was not bugged while I said it was: I had in mind the 1st change, you had in mind the second
BTW that's quite funny to try fixing a loading problem with... A program to load
I will investigate all this ROM/patch story in detail and try to write an article about it.

Now, a poll: this 'ALC' program is useless with Novalight. Should the WAV generator keep it anyway (for historical reasons or whatever), or should it detect it and remove it from the WAV file?

[iss]

For the poll: If it's not too difficult - detect and discard it during WAV generation.
Here is the quick disassembly of ALC:

Code: Select all

LE7B6           := $E7B6
LE867           := $E867
LEE22           := $EE22
        php
        sei
        lda     $FFF9
        cmp     #$01
        bne     L02B2
        lda     $E4B6
        cmp     #$A2
        bne     L02A6
        ldy     #$09
L0293:  lda     L02B5,y
        sta     $0221,y
        dey
        bpl     L0293
        lda     #$21
        sta     $0245
        lda     #$02
        sta     $0246
L02A6:  jmp     LE867

        sta     ($02,x)
        .byte   $BF
        .byte   $02
        .byte   $C7
        .byte   $80
        .byte   $FF
        .byte   $FF
L02B1:  brk
L02B2:  jmp     LE7B6

L02B5:  pha
        lda     #$00
        sta     L02B1
        pla
        jmp     LEE22

It intercepts the IRQ and clears the error flag - so not useful (especially) with Novalight I think.
For historical and archive purposes we can comment the above source and upload it (for instance) on oric.org.

@Chema: I'm interested to have your ROM if possible. Do you know is it factory PROM or custom patched EPROM?
Do you know if it's in the list already here: http://oric.free.fr/ROMS/romident.txt
You can use testfbs testing utility to find ROM's crc32 which I posted here.
BTW, did your friend had chance to try the same testfbs with Jasmin ?

[Symoon]

For the poll: If it's not too difficult - detect and discard it during WAV generation.
Here is the quick disassembly of ALC.
It intercepts the IRQ and clears the error flag - so not useful (especially) with Novalight I think.

This leads to another question: would it be useful to have an option to insert ALC before the "normal speed" Novalight loader? What I don't get yet is if the 1st generation of Atmos who might detect parity errors during the synchro, may in certain conditions have loading error mistakes without ALC, even with a good sound. Maybe this is why Chema has trouble loading it...

Edit: mmmh, I probably tested it on an Atmos with the 1st version of the ROM already (no idea which of my machines have this 1st ROM version), so this is probably useless.

[Symoon]

1st working tests with user memory relocation!
Sadly doesn't help Psychiatric loading - it was using page 1 but I suspect it uses specific loading routines or protection.

Anyway.
Now, removing ALC and optimizing dictionary. I think I need some holidays

[Symoon]

The (now useless) ALC program is now detected and skipped by Novalight!

Seems like French editors (Cobra Soft, Loriciels, Infogrames) enjoyed modifying it to store a part of their protection; in such cases of course the modified ALC program is kept.

[Symoon]

After a rest, I'm back to this dictionary matter.
Reading my existing code again, I see something interesting from my thoughts of 2 years ago.

Once the code considers it has chosen rightly the 14 values to put in the dictionary (wrongly or not actually), it sorts them again in the dictionary to make it optimal in a quite obvious way: you need to re-assign each selected value according to its frequency only. The way you selected them doesn't matter anymore: if you selected 500 "A", 30 "B" and 200 "C", you will obviously minimize the newly inserted length by assigning the shortest time to "A", then "C", then "B".

If you look at your Novalogic table BTW, it seems each time you choose to invert the selection order of two values, the 2nd one actually has more occurencies (is it just by chance, I don't know! Only exception is the last "TRUE" which I actually don't understand why it says "TRUE" ).

You also can see that, from the new signal length point of view, it's obvious that it would be much better to use "I" (frequency 940) from 3rd or 2nd, to 1st position, than K (frequency 860)
=> 940*4+860*6=8920, is better than K 1st and I 2nd: 940*6+860*4=9080

So I wonder if the problem solution could not actually be simplified/optimized:
1- remove the biggest frequency*gain for the first 13 values: if a choice had to be inverted, it will be selected next anyway
2- 14th value has to take the next choice into consideration to select the best one (if you don't, you may drop the best choice, just like my very first example with "A" and "B") EDIT: mmmh, no, this was another problem actually (I was not using gain, but initial length only)
3- the gain is finally optimized by sorting the 14 values by frequency

... Or am I again on a totally wrong track?
EDIT: going to bed, I'm actually a bit lost again in how to initially select the right 14 values! I tend to think we are close to perfection with your initial selection: frequency*gain (which means calculation for each dictionary entry), and with my final optimisation which is just sorting again by freqency the 14 selected items, avoiding this "do I switch my choice or not" step during the selection.

[Symoon]

I tend to think we are close to perfection with your initial selection: frequency*gain (which means calculation for each dictionary entry), and with my final optimisation which is just sorting again by freqency the 14 selected items, avoiding this "do I switch my choice or not" step during the selection.

Ok, just tried this solution, and it's actually better than my previous "use an average value to compute the gain"!
=> Zorgon is 0.01 second faster, choosing $CE (frequency 271), over $05 (frequency 613) with the previous method. That may sound ridiculous, but it just shows it's better, and compensates about 50% of the time lost with the new Novalight structure. I like that
Maybe with other programs the gain will be better.

EDIT: arrrgh, nope, 3D Fongus is slower with this new version. Something is still wrong somewhere. I think I got it: the last value to be chosen must be compared between a gain, and a length..
For instance if I take A and leave B out of the dictionary, I will gain what I calculated for A, but I will leave for B its inital length, not gain. The last comparison is different from the previous ones.

[Symoon]

Ok, after another evening/morning working at it, I realise there is no simplified way to be optimal with the dictionary: each attempt (tried about 8 of them) gave better results on several programs, and inferior results on other. None was "always better".

The only solution IMHO would be to compute and combine all possibilities. Forget it, the gain is too small for such a complexity to deal with
So I will keep the code as it was, avoiding to complexify it too much for a random minor gain. That being said, anyone willing to give it a try as we did with NekoNoNiaow, please do

[Symoon]

Novalight 1.2a just released.
https://sourceforge.net/projects/euphor ... Novalight/

Holds the following changes since previously released v1.1k:
V 1.2a (released 2019-03-24):
- auto-relocation in memory if page 1 is used by the program to load. /!\ heavy ASM/C dependency
- ALC program is now detected and skipped (it's a now useless patch for old tape players)
- public release, cumulating all previous changes from v1.1l to v1.1n.

V1.1n (not released)
!!! incompatibility with ALC program and Loriciels software using it as copy protection!
=> new version required to store the dictionary in page 1.
- stopped using page 2 to store the dictionary, it caused compatibility problems (especially with Loriciels
software). This leads to split Common Area 0 with additional 0a and 0b parts.
- this means the old loader is obsolete, so "old loader" option removed. Same for "no loader", as the old
loader was required to load a Novalight file without loader.
- "multipart booster" option allows to use a reduced Common Area 1 if the Kernel and Common Area 0 are kept
intact. Requires re-loading of Common Area 0a at the end of a part.
- optimized the dictionary by not only taking care of the time wasted by the most frequent isolated
bytes, but also by taking care of how much time they use with their new encoding in the dictionary.
(sometimes, it is worth selecting a byte that occupies a bit less time in the original signal,
but that will occupy much less time with its new encoding!)


V1.1m (not released)
- removed the $24 starting byte. This was here for historical versions and had become useless.
- the room saved by this removal (7 bytes) is now used for a new change: in an uncompressed byte,
the bit sequence '1111', which should be coded by 6 + 6 samples, is now coded by 7 samples.

!!! OLD LOADER not working anymore.

V1.1l (not released)
- removed 3 useless bytes in Common Area 2: in ROM 1.0, clearing the status line is
done by the CLOAD (it was historically required because Novalight used to display
its own "Searching", but this has been removed before the 1st public release)
- added 'multipart booster' option
- corrected a bug: oldloader was ignored if combined with F16 speed
- just for the joke, modified the options letters a bit (-s, -i, -m, -o and -n)

'No Loader' is superior to all the other options: standard speed, old loader and multipart booster will be ignored.
'Old Loader' will ignore 'Multipart Booster', as the booster only works with the normal loader.

[Symoon]

PS: ISS, unless someone finds bugs, I don't plan to change Novalight anymore, not before a while I mean
So you can go for it with TapOric

[iss]

Cool! Congrats for the new release. I'll try to integrate it asap.

[Symoon]

Good luck with the code

[Symoon]

PS: ISS, unless someone finds bugs, I don't plan to change Novalight anymore, not before a while I mean
So you can go for it with TapOric

(oh, well, I've just had a new idea... Might require some testing though... But very tempting... That would make of me a liar )

EDIT: will probably try, though I don't like much the idea of reading a signal with a margin limited to 1 or 2µs.

[NekoNoNiaow]

After a rest, I'm back to this dictionary matter.
[snip]
If you look at your Novalogic table BTW, it seems each time you choose to invert the selection order of two values, the 2nd one actually has more occurencies (is it just by chance, I don't know! Only exception is the last "TRUE" which I actually don't understand why it says "TRUE" ).

If you look at the graph which shows the frequencies of each candidates, this is much clearer.
Choosing K is definitely the best choice as first candidate because it gives the better gain compared to the original size.

You also can see that, from the new signal length point of view, it's obvious that it would be much better to use "I" (frequency 940) from 3rd or 2nd, to 1st position, than K (frequency 860)
=> 940*4+860*6=8920, is better than K 1st and I 2nd: 940*6+860*4=9080

Yes, but what matters is what gives you the best size reduction overall by comparing how much you gain by encoding candidates using the dictionary compared to their original size.

Encoding K first, gives a (140-4)*860 = 116 960 bytes gain over the original non-compressed size.
Encoding I first, gives a (90-4)*940 = 80840 bytes gain over the original non-compressed size.

Even if encoding I first takes less place than encoding K first, this does not matter because encoding K gives a much better size reduction.
K can still take more place than I and have a better size reduction overall.

But even then, check at how the INVERT field is computed:

Code: Select all

=sum(large(D$2:D$16,1),large(E$2:E$16,2))-sum(large(D$2:D$16,2),large(E$2:E$16,1)) < 0

It sums the gains of both possible choices and compares them : K 1st + I 2nd < K 2nd + I 1st.
And the result is : K 1st gives a better gain.

As I mentioned in my previous post, for the subsequent entries, this is less true. But note that the graph is very helpful in visualizing when inverting is better than the greedy " always encodes the candidate with the best gain at each step" strategy.

So I wonder if the problem solution could not actually be simplified/optimized:
1- remove the biggest frequency*gain for the first 13 values: if a choice had to be inverted, it will be selected next anyway

This does not work, there is no general rule that says that inverting is good, this solely depends on the data.
And as you mention in your next post, only exhausting all combinations can tell you what is the best choice.

[Symoon]

To be honest, I tried all the solutions we talked about and always found a program which ended being slower
So after a full day/night spent on it for a difference of something like 0.005s, I ended up choosing to keep a simple code and make a break with this - really needed to.

I'm working on another optimization which, should it work, may save (I hope) about 1 second for Zorgons Revenge. And it changes the dictionary too :p