Loci - my Oric storage emulation project

[xahmol]

Hmm, now see it is actually not exactly the same as I had before:

Code: Select all

Memory region         Used Size  Region Size  %age Used
           FLASH:      166052 B        16 MB      0.99%
             RAM:      197612 B       192 KB    100.51%
       SCRATCH_X:          2 KB         4 KB     50.00%
       SCRATCH_Y:          0 GB         4 KB      0.00%

Now the overflow is only 100.51%, not my earlier 106.13% So it now almost compiles.

Weird.....

[Sodiumlightbaby]

Looks like the cmake build type may not have been correct then (106% vs 100%)?

Might be the build environment then:
I'm on native Ubuntu 24.04.1 LTS
gcc-arm-none-eabi version (15:13.2.rel1-2)
libnewlib-arm-none-eabi version (4.4.0.20231231-2)

[xahmol]

Directory delete via the remove() function works by the way (if the dir is empty that is, but that is desired behaviour).

[xahmol]

Looks like I am somewhat behind in version:

Code: Select all

gcc-arm-none-eabi:
  Installed: 15:10.3-2021.07-4
  
libnewlib-arm-none-eabi:
  Installed: 3.3.0-1.3

Apparently apt update / upgrade does not fetch the newest version then. Also noted that on CC65 actually, reason why I compiled CC65 from source instead of installing the package.
Will investigate later. Probably have to upgrade from Ubuntu 22.04 to 24.04 then. Downside of WSL2 is that release upgrade is not trivial, so need some time to do that (for the rest love WSL2 as it really integrates very well Win11 and Linux on one desktop. Including full drag and drop between Win11 and Linux windows, plus full Linux file system integration in Windows Explorer making drag and drop between Win11 and Linux file folders very easy).

[iss]

This is from the map file where the ASSERTion fails:

Code: Select all

.flash_end  0x10045aac  0x0
            0x10045aac  __flash_binary_end = .
            0x20040000  __StackLimit = (ORIGIN (RAM) + LENGTH (RAM))
            0x20041000  __StackOneTop = (ORIGIN (SCRATCH_X) + LENGTH (SCRATCH_X))
            0x20042000  __StackTop = (ORIGIN (SCRATCH_Y) + LENGTH (SCRATCH_Y))
            0x20040800  __StackOneBottom = (__StackOneTop - SIZEOF (.stack1_dummy))
            0x20041800  __StackBottom = (__StackTop - SIZEOF (.stack_dummy))
            0x20042000  PROVIDE (__stack = __StackTop)
=====>      0x00000000  ASSERT ((__StackLimit >= __HeapLimit), region RAM overflowed)
            0x00000001  ASSERT (((__binary_info_header_end - __logical_binary_start) <= 0x100), Binary info must be in first 256 bytes of the binary)

Interesting is that even RAM usage is <100% link fails (IMHO the ~2k stack is not included in percent calculation):

Code: Select all

/usr/lib/gcc/arm-none-eabi/14.1.0/../../../../arm-none-eabi/bin/ld: region RAM overflowed
Memory region         Used Size  Region Size  %age Used
           FLASH:      268940 B        16 MB      1.60%
             RAM:      194576 B       192 KB     98.97%
       SCRATCH_X:          2 KB         4 KB     50.00%
       SCRATCH_Y:           0 B         4 KB      0.00%
collect2: error: ld returned 1 exit status

I'm using: arm-none-eabi-gcc (Fedora 14.1.0-1.fc40) 14.1.0, probably version diff is involved in the issue too.

Anyway, I found adding more build-in ROMs eats RAM too although ROM arrays are properly set with '__in_flash()'.
And when I removed some now build succeeds with 5x16k + 1x8k ROMs (i.e.: locirom, basic10.rom, basic11b.rom, pravetz11a.rom, test108k.rom and microdis.rom) .
(raxiss.com updated.)

[xahmol]

Thanks @iss again!

And mkdir() works. Great!

Only found that mkdir() actually is quite slow compared to other operations and is not finished doing stuff on the file system when the function returns.
First thought it was not working as parsing the dir again did not show the dir, but refreshing dir and voila there it was.
Then added some waiting time, and then the redraw of the dir clearly got the dir creation midway as the dirname was only half created, giving funny results.

Now added a waiting loop that loops until the new dir can be opened:

Code: Select all

                // Wait for dir create to finish
                while (dir = opendir(pathbuffer), !dir)
                {
                    // Wait
                    ;
                }
                closedir(dir);

Would this be really the right way to do it? Or is there some other way to wait until LOCI file IO internally is done?
Of course that would be crucial if you want to directly save files to a newly created dir from a program, so I would even suggest to include 'check if done' in the _sysmkdir() function itself. Otherwise any disk IO to the new dit directly after dir creation might fail, or worse, have unexpected results.

Edit: And now also realize what might be an issue with my present 'try to open dir' loop: this would give an endless loop if for example the user janks the USB out during this loop. So propably not the most ideal way to do it.

Edit2: Also tried on internal flash, and then dir creation is almost instant. Did not try, but do not think the wait loop is even needed in that case.

All the dirs test* are newly created.

[xahmol]

Oh, another strange issue:

Dir deletion after dir creation works great on USB drive. My routine failed on deleting a newly created dir on internal flash.
Why? Fount that on creating a dir on internal flash it creates a . subdir in that new dir. Did not check, but probably also a .. subdir, just like are present in the root.

I filter them out in dir parsing, so first was not understanding why my 'check if dir is empty' routine failed. But see it if I print debug info:
See the 10 . on the third line: 10 is the attribute in hex, so DIR, . is the name.

Is this a bug? Or something I have to live with and account for by also deleting the . and .. subdirs on dir delete?
Or is there any way to force a recursive delete?

Edit: Found that actually if I comment out my 'check if dir is empty' the dir delete on internal flash actually does work. While it fails as it should if I place actual files in. So for the internal storage filesystem, a dir with those . and .. subdirs is actually considered empty.
So workaround: I will not only check if dir is empty for USB, and just let it fail with OS error if internal storage dir is not empty on delete attempt.

Or is there a better way, working on both USB as internal flash, to check if dir is empty than this?

Code: Select all

        // Check if dir is empty
        // First: open dir
        dir = opendir(pathbuffer);
        if (!dir)
        {
            // Exit if dir can not be opened
            menu_fileerrormessage();
            return;
        }

        // Then: try to read first entry
        file = readdir(dir);
        if (file->d_name[0] != 0)
        {
            // Exit if dir is not empty
            menu_messagepopup("Directory is not empty.");
            closedir(dir);
            return;
        }
        // Close dir
        closedir(dir);

Full code and latest build still here. Needs newer firmware build than on the official release yet, latest Raxiss build will work. Do not yet check for FW version (also because uname gives the same 0.2.5 version also for the newest Raxiss build), assume not implemented opp codes just do nothing? Or give OS error?
https://github.com/xahmol/locifilemanager

[Sodiumlightbaby]

Only found that mkdir() actually is quite slow compared to other operations and is not finished doing stuff on the file system when the function returns.

One thing to remember for the experts here, is that the file and directory operations we expose on the LOCI API are thin layers on top of either the LittleFS system on internal flash or the FatFS system on USB devices. Experienced differences between internal flash and USB will be due to this and to the different physical layers.

In this case it is either the FatFS not synching after creation, or a weakness in the flushing of the TinyUSB stack. I suspect the latter. This is probably fixable, at some point.

Fount that on creating a dir on internal flash it creates a . subdir in that new dir. Did not check, but probably also a .. subdir, just like are present in the root.

Is this a bug? Or something I have to live with and account for by also deleting the . and .. subdirs on dir delete?

Well, this is a feature of the LittleFS lfs_read_dir() function, but might be considered a bug for LOCI. There are no such created directories, they just provide these references when you do a directory listing on the internal flash. This is probably trivial to fix in the LOCI API shim in the firmware.

[xahmol]

One thing to remember for the experts here, is that the file and directory operations we expose on the LOCI API are thin layers on top of either the LittleFS system on internal flash or the FatFS system on USB devices. Experienced differences between internal flash and USB will be due to this and to the different physical layers.

Gathered as much.
Just like that I realised after posting that the . and .. are actually of course present in all *nix like file systems, therefore the cd .. and ./file
Just never actually encountered them this way, but then again, never wrote del dir functions from scratch either

I assume then that an opendir to 0:/test/sub/.. would bring you actually to 0:/test/ as parent?
That would explain some weird behaviour I had when not filtering . and .. out yet and trying to dir change to those…..
Rather confused my full path building logic that does not do getcwd every time (mainly because when I wrote this part this function did not even exist yet in the firmware) but builds the path from root as starting point.

[Sodiumlightbaby]

Only found that mkdir() actually is quite slow compared to other operations and is not finished doing stuff on the file system when the function returns.

I had a look at the code for this, and the FatFS side of things looks ok. It does blocking writes to the TinyUSB MSC code, so the use of the TinyUSB functions also look good, as it reports all writes as completed. So current leading candidate (if I'm not badly mistaken) is low-level synchronisation issue in the TinyUSB stack. I'm trying to bring it up to the latest version to see if it helps.

[xahmol]

Question:
On your firmware versioning, will major, minor, patch be always between one digit 0-9? Or might patch level of minor reach double digit somewhere?

Think I want version detection (to be able to decide to give different functionality based on FW version) and convert to a version struct with unsigned char major, minor and patch.
And of course parsing the uname string is much more easy if the version string in uname is always one digit one dot one digit one dot one digit.

(For example: if my program needs mkdir() it needs to check that FW version I assume to be 0.2.6 is there. Of course checking for versions prior to 0.2.5 is not possible as it relies on uname introduced in that version. But as also the L marker is not there prior to 0.2.5, my program simply will not start and only give the message No Loci present. Might actually change that to ‘No LOCI or old firmware’ therefore)

[Sodiumlightbaby]

Question:
On your firmware versioning, will major, minor, patch be always between one digit 0-9? Or might patch level of minor reach double digit somewhere?

We did go quite high on the 0.1.x development so I think it is safe to say we do double digits. What we can become more strict with is incrementing minor when there is an API change, so you should only need to work with major & minor. Yes it would mean requiring 0.2.x users to upgrade even if _some_ 0.2.x release could be supported. Does that work for you?

[xahmol]

Yes, certainly works.
Could do double digit also, but gives a lot more code especially in target binary as strol() is not a small function.

[xahmol]

Have (at least for now) chosen to do it this way:

Code: Select all

    // Set version number struct
    locicfg.version.major = locicfg.uname.release[0] - '0';
    locicfg.version.minor = locicfg.uname.release[2] - '0';
    locicfg.version.patch = locicfg.uname.release[4] - '0';
    if (locicfg.uname.release[5])
    {
        locicfg.version.patch = locicfg.version.patch * 10 + locicfg.uname.release[5] - '0';
    }

Assume your patch version numbers will at least be not more than two digits

[Dbug]

If the versions are stored in BCD, you can have xx.yy.zz stored in only 3 bytes with each value having two digits