OSDK code generation bug

Questions, bug reports, features requests, ... about the Oric Software Development Kit. Please indicate clearly in the title the related element (OSDK for generic questions, PictConv, FilePack, XA, Euphoric, etc...) to make it easy to locate messages.

User avatar
Dbug
Site Admin
Posts: 4444
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

OSDK code generation bug

Post by Dbug »

Hello,
so yesterday I digged a bit in the issue related to the OSDK generating invalid code when adding some code in a C program.

What I did, is to add parameters to the program, in a function call. And I immediately experienced code not working as planned. The interesting thing being that the new function was not even being called, just having the code present was enough to cause the issue.

So it appears that for some reason, the code bellow is corrupting data, the only reason I can find is an issue with the stack management (stack being at the end).

Since it may take some time before I find the problem, I decided to explain how the whole thing work.

1) The project

I'm going to take as an example the code from Barnsey, located in the SVN server in "users/barnsey123". My SVN depot is mapped on E:\SVN, so in the E:\svn\users\barnsey123 folder I have:

- main.c
- osdk_build.bat
- osdk_config.bat
- osdk_execute.bat

When I run the osdk_build.bat a BUILD folder appear, with the following content:

- final.out
- symbols
- viking.tap
- xaerr.txt

When I run the osdk_execute.bat my emulator is started, and it loads viking.tap. The question is, what happens in-between that leads to the transformation of main.c in viking.tap?

2) The build process

The important parameters are set in the osdk_config.bat:

Code: Select all

SET OSDKADDR=$800
SET OSDKNAME=VIKING
SET OSDKFILE=MAIN
When osdk_build.bat is called, it will use these parameters to convert the list of source files (specified in OSDKFILE) to an Oric executable file called as what is specified in OSDKNAME. The build process itself is done by make.bat, located in the main OSDK distribution folder at %OSDK%\bin\make.bat.

Make.bat will process each file, and based on the extension found in the source folder, will treat it as a BASIC file (.bas), ASSEMBLER file (.s) or C file (.c).

- The BASIC files are converted using BAS2TAP
- The ASSEMBLER files are directly copied to the TMP folder
- The C files are compiled to assembler and stored in the TMP folder
Then at the end the various files are linked together, and assembled by XA to generate the final binary executable.

In this particular case, we have only one file, MAIN.C. Let's see how it's converted from C to 6502 assembler.

3) The C compiler

If you build the program, and then take a look in %osdk%\TMP you will find the following files:
- MAIN.c
- MAIN.c2
- MAIN.s
- MAIN

plus two files we will talk about later:
- link.bat
- linked.s

The first file, MAIN.c is still a C program, but it contains the pre-processed content of the original main.c. This means that there are no more #include commands (the files have been inlined in the main source file), there are no more comments (they have been removed), and every single #define parameter has been replaced by its real value.

So it's still C, but without any external dependencies.

So in this particular case, it converted the original drawbox function from:

Code: Select all

// DRAWBOX - draw a box at x,y length xbox
void drawbox(int x, int y, int z)
{
	/*
	x=xcoord for curset
	y=ycoord for curset
	z=size of line to draw
	*/
	curset(x,y,1);
	draw(z,0,1);
	draw(0,z,1);
	draw(-z,0,1);
	draw(0,-z,1);
}
to this filtered out version:

Code: Select all

# 8 "MAIN.c" 2

 
void drawbox(int x, int y, int z)
{
	 




	curset(x,y,1);
	draw(z,0,1);
	draw(0,z,1);
	draw(-z,0,1);
	draw(0,-z,1);
}
Please note that the empty lines are kept in order to allow the C compiler to report the errors at the correct location in the original file.

The second file, MAIN.c2 contains the results of the compilation. It's not 6502 assembler, it's using an intermediate representation using a weird syntax. Our drawbox function looks like this now:

Code: Select all

_drawbox
	ENTER(1,6)
	INDIRW_AD((ap),4,tmp0)
	ASGNW_DC(tmp0,reg0)
	INDIRW_AD((ap),0,tmp0)
	ARGW_D(tmp0,(sp),0)
	INDIRW_AD((ap),2,tmp0)
	ARGW_D(tmp0,(sp),2)
	ARGW_C(1,(sp),4)
	CALLV_C(_curset,6)
	INDIRW_CD(reg0,tmp0)
	ARGW_D(tmp0,(sp),0)
	ARGW_C(0,(sp),2)
	ARGW_C(1,(sp),4)
	CALLV_C(_draw,6)
	ARGW_C(0,(sp),0)
	INDIRW_CD(reg0,tmp0)
	ARGW_D(tmp0,(sp),2)
	ARGW_C(1,(sp),4)
	CALLV_C(_draw,6)
	INDIRW_CD(reg0,tmp0)
	NEGI_DD(tmp0,tmp0)
	ARGW_D(tmp0,(sp),0)
	ARGW_C(0,(sp),2)
	ARGW_C(1,(sp),4)
	CALLV_C(_draw,6)
	ARGW_C(0,(sp),0)
	INDIRW_CD(reg0,tmp0)
	NEGI_DD(tmp0,tmp0)
	ARGW_D(tmp0,(sp),2)
	ARGW_C(1,(sp),4)
	CALLV_C(_draw,6)
	LEAVE
I guess at this point you are wondering how we go from this weird language to the final 6502 assembler? Well, for that we are calling the preprocessor a second time, passing as a parameter the content of the %OSDK%/macro/MACROS.H file.

Here is an extract of this file showing the implementation of the ENTER virtual instructions:

Code: Select all

#define ENTER(r,n)\
	ldx #n ;\
	lda #r ;\
	jsr enter ;\
I guess that's self explanatory. Each instruction is implemented as a macro, which is then expended in as many 6502 instructions as necessary.

After going through the preprocessor we obtain MAIN.S. Here is our final drawbox assembler code:

Code: Select all

_drawbox
	ldx #6 ;	lda #1 ;	jsr enter ;
	ldy #4 ;	lda (ap),y ;	sta tmp0 ;	iny ;	lda (ap),y ;	sta tmp0+1 ;
	lda tmp0 ;	sta reg0 ;	lda tmp0+1 ;	sta reg0+1 ;
	ldy #0 ;	lda (ap),y ;	sta tmp0 ;	iny ;	lda (ap),y ;	sta tmp0+1 ;
	lda tmp0 ;	ldy #0 ;	sta (sp),y ;	iny ;	lda tmp0+1 ;	sta (sp),y ;
	ldy #2 ;	lda (ap),y ;	sta tmp0 ;	iny ;	lda (ap),y ;	sta tmp0+1 ;
	lda tmp0 ;	ldy #2 ;	sta (sp),y ;	iny ;	lda tmp0+1 ;	sta (sp),y ;
	lda #<(1) ;	ldy #4 ;	sta (sp),y ;	iny ;	lda #>(1) ;	sta (sp),y ;
	ldy #6 ;	jsr _curset ;
	lda reg0 ;	sta tmp0 ;	lda reg0+1 ;	sta tmp0+1 ;
	lda tmp0 ;	ldy #0 ;	sta (sp),y ;	iny ;	lda tmp0+1 ;	sta (sp),y ;
	lda #<(0) ;	ldy #2 ;	sta (sp),y ;	iny ;	lda #>(0) ;	sta (sp),y ;
	lda #<(1) ;	ldy #4 ;	sta (sp),y ;	iny ;	lda #>(1) ;	sta (sp),y ;
	ldy #6 ;	jsr _draw ;
	lda #<(0) ;	ldy #0 ;	sta (sp),y ;	iny ;	lda #>(0) ;	sta (sp),y ;
	lda reg0 ;	sta tmp0 ;	lda reg0+1 ;	sta tmp0+1 ;
	lda tmp0 ;	ldy #2 ;	sta (sp),y ;	iny ;	lda tmp0+1 ;	sta (sp),y ;
	lda #<(1) ;	ldy #4 ;	sta (sp),y ;	iny ;	lda #>(1) ;	sta (sp),y ;
	ldy #6 ;	jsr _draw ;
	lda reg0 ;	sta tmp0 ;	lda reg0+1 ;	sta tmp0+1 ;
	lda #0 ;	sec ;	sbc tmp0 ;	sta tmp0 ;	lda #0 ;	sbc tmp0+1 ;	sta tmp0+1 ;
	lda tmp0 ;	ldy #0 ;	sta (sp),y ;	iny ;	lda tmp0+1 ;	sta (sp),y ;
	lda #<(0) ;	ldy #2 ;	sta (sp),y ;	iny ;	lda #>(0) ;	sta (sp),y ;
	lda #<(1) ;	ldy #4 ;	sta (sp),y ;	iny ;	lda #>(1) ;	sta (sp),y ;
	ldy #6 ;	jsr _draw ;
	lda #<(0) ;	ldy #0 ;	sta (sp),y ;	iny ;	lda #>(0) ;	sta (sp),y ;
	lda reg0 ;	sta tmp0 ;	lda reg0+1 ;	sta tmp0+1 ;
	lda #0 ;	sec ;	sbc tmp0 ;	sta tmp0 ;	lda #0 ;	sbc tmp0+1 ;	sta tmp0+1 ;
	lda tmp0 ;	ldy #2 ;	sta (sp),y ;	iny ;	lda tmp0+1 ;	sta (sp),y ;
	lda #<(1) ;	ldy #4 ;	sta (sp),y ;	iny ;	lda #>(1) ;	sta (sp),y ;
	ldy #6 ;	jsr _draw ;
	jmp leave ;
The final file, MAIN is just the same, but with the carriage return fixed to avoid having multiple instructions on the same line.

4) The linker

The final executable source code is generate by the linker, which collects all the converted source codes, find all the external references (function calls like printf or get) referenced in the %osdk%/library.ndx file, and finally adds the header.s and tail.s (in the same folder) to provide support for the C stack and stack frame management.

The result of the linker's work is the %osdk%/tmp/linked.s, which is then sent to XA.EXE for finally creating the final binary.

Now that we've explained everything, let's take a look at the problem.

5) The problem

And easy way to visualise the problem is to modify the program in the following way.

Modify readkeyb from this:

Code: Select all

void readkeyb()	// filter keyboard input
{
	int key=0;
	int cx=98;
	int cy=86;
to that:

Code: Select all

void readkeyb(int cx,int cy)	// filter keyboard input
{
	int key=0;
and then later change

Code: Select all

	readkeyb();
to

Code: Select all

	readkeyb(98,86);
So basically instead of hard-coding the parameters in the readkeyb function we pass them as parameters.

Just by doing that the program stop drawing things correctly. The board only shows one single tile on the top left.

The interesting thing is that the readkeyb function is called AFTER the drawboard, so the modified code is never actually called.

The second thing, is that if you comment out the call to readkeyb, then the program runs fine.

The culprit at this point seems to be the compiler which probably generates wrong code (after all the whole process is very convoluted, no?) but if you compare both linked.s files, the only difference in the generate assembly code is the missing lines that correspond to the call of readkeyb that we just commented out:

Code: Select all

	lda #<(98) 
	ldy #0 
	sta (sp),y 
	iny 
	lda #>(98) 
	sta (sp),y 

	lda #<(86) 
	ldy #2 
	sta (sp),y 
	iny 
	lda #>(86) 
	sta (sp),y 

	ldy #4 
	jsr _readkeyb 
Everything else is absolutely identical between the two versions. Except the one with the readkeyb call fails while the other works.

At this point the only idea I have is that for some reason the added code is triggering some issue in the addresses/function calls, possibly because the Enter/Leave code is at the top of the executable while the stack is at the very end.

Stay tuned for more information, and please don't hesitate sharing your thoughts.
User avatar
Dbug
Site Admin
Posts: 4444
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Post by Dbug »

I managed to reproduce the problem without changing any code, just by playing with the number of characters in the "printf string".

When looking at the compiled code and the symbol files, the offsets are all correctly computed, nothing is corrupted.

The only conclusion I can reach from that, is that the bug is in the libraries, probably in the header.s file. Would not be surprised if it was some error in the code when computing the address of the stack frame.
User avatar
barnsey123
Flight Lieutenant
Posts: 379
Joined: Fri Mar 18, 2011 10:04 am
Location: Birmingham

Post by barnsey123 »

Hi Dbug, what you describe tallies exactly with my experiences. I've re-written this program several times changing all manner of things.
Printf did seem to upset things. I was trying to find out the cause of the problem by printing variables at different locations but trying to see the values changed the values! (Like some weird Quantum effect)

One idea related to lib.h (this contains amongst other things "printf" but that is also contained in stdio.h) I'm wondering if lib.h is the culprit by not accurately duplicating the standard functions. I found this out when trying to include stdio.h AND lib.h in the same program (it complains about duplicate functions). Hope this helps. :D
User avatar
barnsey123
Flight Lieutenant
Posts: 379
Joined: Fri Mar 18, 2011 10:04 am
Location: Birmingham

Post by barnsey123 »

This is what happens when INCLUDING lib.h and stdio.h
printf and putchar are duplicated. I just wonder if something isn't quite right in the lib.h versions

Building the program VIKING at adress $800
Compiling MAIN.C
- preprocess
- compile
MAIN.c: C:\\oricutron\\osdk_0_017\\include\\stdio.h(13): redeclaration of `putch
ar' previously declared at C:\\oricutron\\osdk_0_017\\include\\lib.h(5)
MAIN.c: C:\\oricutron\\osdk_0_017\\include\\stdio.h(48): redeclaration of `print
f' previously declared at C:\\oricutron\\osdk_0_017\\include\\lib.h(6)
ERROR : Build failed.
Press any key to continue . . .
User avatar
Dbug
Site Admin
Posts: 4444
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Post by Dbug »

barnsey123 wrote:One idea related to lib.h (this contains amongst other things "printf" but that is also contained in stdio.h) I'm wondering if lib.h is the culprit by not accurately duplicating the standard functions. I found this out when trying to include stdio.h AND lib.h in the same program (it complains about duplicate functions). Hope this helps. :D
Not sure, I personally never include stdio.h on the Oric, mostly because we do not have any library for the standard input/ouput, so kind of defeat the purpose :)

That being said, I think I may have found the issue, it's kind of pervert and I need to check, but I suspect this part of the code (in header.s):

Code: Select all

	iny
	lda (fp),y
	sta fp 
	iny
	lda (fp),y
	sta fp+1
A possibility that came to my mind, is that the sta fp modified the fp which is used by the lda (fp),y, so in the case where we cross a page boundary, the lda will suddenly loads from the wrong place, which can result in interesting things.
User avatar
barnsey123
Flight Lieutenant
Posts: 379
Joined: Fri Mar 18, 2011 10:04 am
Location: Birmingham

Post by barnsey123 »

Let me know if you need me to test anything (there's a whole bunch of C code I need to write so any beta release of an amended OSDK will get a thorough shakedown). :D
User avatar
Dbug
Site Admin
Posts: 4444
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Post by Dbug »

Ok, I think I got it fixed:

- edit %osdk%/lib/header.s
- locate the leave function

- replace this code:

Code: Select all

	iny
	lda (fp),y
	iny
	sta fp
	lda (fp),y
	sta fp+1
by that one:

Code: Select all

	iny
	lda (fp),y
	tax
	iny
	lda (fp),y
	sta fp+1
	stx fp
If it's working fine, I will publish an updated version of the OSDK.
User avatar
barnsey123
Flight Lieutenant
Posts: 379
Joined: Fri Mar 18, 2011 10:04 am
Location: Birmingham

Looking Good!

Post by barnsey123 »

Dbug,

have made the suggested changes in header.s and it looks like everything is working perfectly! :D :D :D

I've made all sorts of changes and everything works as expected.

Thanks for the swift resolution. Excellent!
User avatar
ibisum
Wing Commander
Posts: 1646
Joined: Fri Apr 03, 2009 8:56 am
Location: Vienna, Austria
Contact:

Post by ibisum »

Is there any chance we can get the OSDK built for Linux/Mac OSX somewhere along the lines? Maybe?
User avatar
Dbug
Site Admin
Posts: 4444
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Post by Dbug »

Well, the problem is that all that takes time, and when it's not maintained then it breaks.

The full source code is available, so it's possible to make it work on Linux or Mac/OS as well.

As long as I can still package the Windows version without any problem, I'm fine with people making changes. My "making changes", I imply "the changes necessary to make it work", not "random changes of style because my style does not please other people" :)

If somebody wants to do radical changes, there's no problem with forking the whole SDK. It's pretty much usable these days, I'm not doing many changes, and the few changes I would do would be either incremental fixes or additional features in tools, so technically these changes could be imported in a forked version as well.

And if fork there is, I'm find with hosting the forked versions on the same SVN depot, as long as they are clearly identified to make it clear why this fork was done, and what features/system it targets.

The rationale behind that, is that most of the ease of use of the OSDK comes from the fact that it uses batch files. There are no such thing as a standard build system on Windows, so you can't just replace osdk_build/execute by some makefiles, that would force people to have to install a lot of software just for being able to compile an Oric program.

On the other hand, I understand that on Unix derived systems, there's already a well implemented set of tools that could be used to do the equivalent using native things.

Porting the executables that form the OSDK is easy, porting the concept of the OSDK oric programs with the config and it builds everywhere is way less obvious :)
User avatar
tingo
Pilot Officer
Posts: 68
Joined: Sat Jul 11, 2009 11:30 am
Location: Oslo, Norway

Post by tingo »

Dbug wrote: The full source code is available, so it's possible to make it work on Linux or Mac/OS as well.
Interesting. I see that it is available via svn. How do I check out / get a copy of the source?
(No, I'm not very familiar with svn).
Torfinn
User avatar
Dbug
Site Admin
Posts: 4444
Joined: Fri Jan 06, 2006 10:00 pm
Location: Oslo, Norway
Contact:

Post by Dbug »

It depends if you use the command line client or a GUI like Subversion.

The front page of the depot gives access to a number of options, including the command line operation you can run to get the depot:

http://miniserve.defence-force.org/svn/readme.htm

If you want to check out the repository with the command-line client:
svn co http://miniserve.getmyip.com/svn
User avatar
tingo
Pilot Officer
Posts: 68
Joined: Sat Jul 11, 2009 11:30 am
Location: Oslo, Norway

Post by tingo »

Dbug wrote:It depends if you use the command line client or a GUI like Subversion.

The front page of the depot gives access to a number of options, including the command line operation you can run to get the depot:

http://miniserve.defence-force.org/svn/readme.htm

If you want to check out the repository with the command-line client:
svn co http://miniserve.getmyip.com/svn
Well, I was hoping for a command that would just give me the source for osdk, not the whole depot.
I use the command line client.
Torfinn
Post Reply