YetAnotherForum
Welcome Guest Search | Active Topics | Log In | Register

Unable to compile on gcc with SQUNICODE defined
Options
Previous Topic Next Topic
maestro
#1 Posted : Sunday, January 22, 2012 5:51:31 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/22/2012(UTC)
Posts: 1

Thanks: 0 times
Was thanked: 0 time(s) in 0 post(s)
Hi.

Let me first note, that I've seen a similar topic on this forum on the same matter. I felt it didn't propose any reasonable answer to the problem, therefore I'm opening a new thread.

I'm in the process of making a cross platform game engine, using squirrel as scripting language, and Sqrat to bind it. I'm not able to compile squirrel in such a way that it uses wchar_t for strings on Linux using gcc version 4.6.1. It works just like expected on windows using VS2010.

When adding the SQUNICODE define to the Makefile in SQUIRREL3/squirrel/Makefile,

Code:

gcc -O2 -DSQUNICODE -fno-exceptions -fno-rtti -Wall -fno-strict-aliasing -c $(SRCS) $(INCZ) $(DEFS)


the compiler reports several errors (amongst others) regarding squirrel.h line 121:

Code:

typedef unsigned short wchar_t;


Upon commenting out this line, the rest of the errors were related to wide string functions, such as swprintf, wcsstr and wcstod.
I was under the assumption that these were caused by missing #include <wchar.t>. I then added this to sqpcheader.h, butthis still produced errors such as

Code:

sqapi.cpp: In function ‘SQInteger sq_aux_invalidtype(HSQUIRRELVM, SQObjectType)’:
sqapi.cpp:37:84: error: invalid conversion from ‘const wchar_t*’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
/usr/include/x86_64-linux-gnu/bits/wchar2.h:286:1: error:   initializing argument 2 of ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’ [-fpermissive]


which from what I can see is due to missing the size parameter in the function call.

Other errors such as

Code:

sqlexer.cpp:258:27: error: ‘iswdigit’ was not declared in this scope
sqlexer.cpp:262:32: error: ‘iswalpha’ was not declared in this scope
sqlexer.cpp:268:26: error: ‘iswcntrl’ was not declared in this scope


were corrected by adding #include <wctype.h> to sqpcheader.h.

The remaining compile log in its entirety:

Code:

gcc -O2 -DSQUNICODE -fno-exceptions -fno-rtti -Wall -fno-strict-aliasing -c sqapi.cpp sqbaselib.cpp sqfuncstate.cpp sqdebug.cpp sqlexer.cpp sqobject.cpp sqcompiler.cpp sqstate.cpp sqtable.cpp sqmem.cpp sqvm.cpp sqclass.cpp -I../include -I. -Iinclude
sqapi.cpp: In function ‘SQInteger sq_aux_invalidtype(HSQUIRRELVM, SQObjectType)’:
sqapi.cpp:37:84: error: invalid conversion from ‘const wchar_t*’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
/usr/include/x86_64-linux-gnu/bits/wchar2.h:286:1: error:   initializing argument 2 of ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’ [-fpermissive]
sqbaselib.cpp: In function ‘bool str2num(const SQChar*, SQObjectPtr&)’:
sqbaselib.cpp:26:45: error: ‘_wcstoi64’ was not declared in this scope
sqdebug.cpp: In member function ‘void SQVM::Raise_Error(const SQChar*, ...)’:
sqdebug.cpp:62:72: error: invalid conversion from ‘const SQChar* {aka const wchar_t*}’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
sqdebug.cpp:62:72: error: cannot convert ‘__va_list_tag*’ to ‘const wchar_t*’ for argument ‘3’ to ‘int vswprintf(wchar_t*, size_t, const wchar_t*, __va_list_tag*)’
sqdebug.cpp: In member function ‘SQString* SQVM::PrintObjVal(const SQObjectPtr&)’:
sqdebug.cpp:77:69: error: invalid conversion from ‘const wchar_t*’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
/usr/include/x86_64-linux-gnu/bits/wchar2.h:286:1: error:   initializing argument 2 of ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’ [-fpermissive]
sqdebug.cpp:77:69: error: invalid conversion from ‘SQInteger {aka long long int}’ to ‘const wchar_t*’ [-fpermissive]
/usr/include/x86_64-linux-gnu/bits/wchar2.h:286:1: error:   initializing argument 3 of ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’ [-fpermissive]
sqdebug.cpp:81:65: error: invalid conversion from ‘const wchar_t*’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
sqdebug.cpp:81:65: error: cannot convert ‘const SQFloat {aka const float}’ to ‘const wchar_t*’ for argument ‘3’ to ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’
sqcompiler.cpp: In member function ‘void SQCompiler::Error(const SQChar*, ...)’:
sqcompiler.cpp:92:25: error: invalid conversion from ‘const SQChar* {aka const wchar_t*}’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
sqcompiler.cpp:92:25: error: cannot convert ‘__va_list_tag*’ to ‘const wchar_t*’ for argument ‘3’ to ‘int vswprintf(wchar_t*, size_t, const wchar_t*, __va_list_tag*)’
sqvm.cpp: In member function ‘bool SQVM::ToString(const SQObjectPtr&, SQObjectPtr&)’:
sqvm.cpp:278:60: error: invalid conversion from ‘const wchar_t*’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
sqvm.cpp:278:60: error: cannot convert ‘const SQFloat {aka const float}’ to ‘const wchar_t*’ for argument ‘3’ to ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’
sqvm.cpp:281:67: error: invalid conversion from ‘const wchar_t*’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
/usr/include/x86_64-linux-gnu/bits/wchar2.h:286:1: error:   initializing argument 2 of ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’ [-fpermissive]
sqvm.cpp:281:67: error: invalid conversion from ‘SQInteger {aka long long int}’ to ‘const wchar_t*’ [-fpermissive]
/usr/include/x86_64-linux-gnu/bits/wchar2.h:286:1: error:   initializing argument 3 of ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’ [-fpermissive]
sqvm.cpp:284:61: error: invalid conversion from ‘const wchar_t*’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
sqvm.cpp:284:61: error: too few arguments to function ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’
/usr/include/x86_64-linux-gnu/bits/wchar2.h:286:1: note: declared here
sqvm.cpp:303:91: error: invalid conversion from ‘const wchar_t*’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
/usr/include/x86_64-linux-gnu/bits/wchar2.h:286:1: error:   initializing argument 2 of ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’ [-fpermissive]
make: *** [sq32] Error 1
maestro@frosty:~/Downloads/sq/SQUIRREL3/squirrel$ make > log
sqapi.cpp: In function ‘SQInteger sq_aux_invalidtype(HSQUIRRELVM, SQObjectType)’:
sqapi.cpp:37:84: error: invalid conversion from ‘const wchar_t*’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
/usr/include/x86_64-linux-gnu/bits/wchar2.h:286:1: error:   initializing argument 2 of ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’ [-fpermissive]
sqbaselib.cpp: In function ‘bool str2num(const SQChar*, SQObjectPtr&)’:
sqbaselib.cpp:26:45: error: ‘_wcstoi64’ was not declared in this scope
sqdebug.cpp: In member function ‘void SQVM::Raise_Error(const SQChar*, ...)’:
sqdebug.cpp:62:72: error: invalid conversion from ‘const SQChar* {aka const wchar_t*}’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
sqdebug.cpp:62:72: error: cannot convert ‘__va_list_tag*’ to ‘const wchar_t*’ for argument ‘3’ to ‘int vswprintf(wchar_t*, size_t, const wchar_t*, __va_list_tag*)’
sqdebug.cpp: In member function ‘SQString* SQVM::PrintObjVal(const SQObjectPtr&)’:
sqdebug.cpp:77:69: error: invalid conversion from ‘const wchar_t*’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
/usr/include/x86_64-linux-gnu/bits/wchar2.h:286:1: error:   initializing argument 2 of ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’ [-fpermissive]
sqdebug.cpp:77:69: error: invalid conversion from ‘SQInteger {aka long long int}’ to ‘const wchar_t*’ [-fpermissive]
/usr/include/x86_64-linux-gnu/bits/wchar2.h:286:1: error:   initializing argument 3 of ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’ [-fpermissive]
sqdebug.cpp:81:65: error: invalid conversion from ‘const wchar_t*’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
sqdebug.cpp:81:65: error: cannot convert ‘const SQFloat {aka const float}’ to ‘const wchar_t*’ for argument ‘3’ to ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’
sqcompiler.cpp: In member function ‘void SQCompiler::Error(const SQChar*, ...)’:
sqcompiler.cpp:92:25: error: invalid conversion from ‘const SQChar* {aka const wchar_t*}’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
sqcompiler.cpp:92:25: error: cannot convert ‘__va_list_tag*’ to ‘const wchar_t*’ for argument ‘3’ to ‘int vswprintf(wchar_t*, size_t, const wchar_t*, __va_list_tag*)’
sqvm.cpp: In member function ‘bool SQVM::ToString(const SQObjectPtr&, SQObjectPtr&)’:
sqvm.cpp:278:60: error: invalid conversion from ‘const wchar_t*’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
sqvm.cpp:278:60: error: cannot convert ‘const SQFloat {aka const float}’ to ‘const wchar_t*’ for argument ‘3’ to ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’
sqvm.cpp:281:67: error: invalid conversion from ‘const wchar_t*’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
/usr/include/x86_64-linux-gnu/bits/wchar2.h:286:1: error:   initializing argument 2 of ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’ [-fpermissive]
sqvm.cpp:281:67: error: invalid conversion from ‘SQInteger {aka long long int}’ to ‘const wchar_t*’ [-fpermissive]
/usr/include/x86_64-linux-gnu/bits/wchar2.h:286:1: error:   initializing argument 3 of ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’ [-fpermissive]
sqvm.cpp:284:61: error: invalid conversion from ‘const wchar_t*’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
sqvm.cpp:284:61: error: too few arguments to function ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’
/usr/include/x86_64-linux-gnu/bits/wchar2.h:286:1: note: declared here
sqvm.cpp:303:91: error: invalid conversion from ‘const wchar_t*’ to ‘size_t {aka long unsigned int}’ [-fpermissive]
/usr/include/x86_64-linux-gnu/bits/wchar2.h:286:1: error:   initializing argument 2 of ‘int swprintf(wchar_t*, size_t, const wchar_t*, ...)’ [-fpermissive]
make: *** [sq32] Error 1


As this is a 64-bit OS, I also tried passing the -m32 flag to gcc in the makefile, but this only changed the include paths of files such as wchar2.h to its 32-bit equivalent, leaving the same errors as before. I don't belive this is the cause, but I have yet to confirm that.

Since most of these functions are defined through macros, it might be best to add another abstraction to them, so if gcc is used to compile, the string operations use an additional size parameter that can be left out on other platforms.

I really hope it's possible to provide a patch for this. I could do it myself, but I'm not proficient enough in making diffs and I'd rather have someone with the full picture provide a fix since these operations seems rather vital to make squirrel work correctly. It might also just be something I'm completely missing or misunderstanding, even though I've done some extensive googling on this matter.

- Maestro
Back to top
montonero
#2 Posted : Thursday, January 31, 2013 2:43:27 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/31/2013(UTC)
Posts: 2

Thanks: 0 times
Was thanked: 0 time(s) in 0 post(s)
I have experienced the same problem. Not using wchars at the moment, but will have to patch squirrel later once we start using them.
Back to top
fagiano
#3 Posted : Friday, February 01, 2013 6:25:44 AM(UTC)
Rank: Advanced Member

Groups: Registered, Administrators
Joined: 6/11/2005(UTC)
Posts: 825

Thanks: 0 times
Was thanked: 36 time(s) in 29 post(s)
unfortunately unicode builds right now only work on Windows, I don't think would be too crazy to fix this problem. GCC just missed( or calls differently) a few functions (plus the wchar_t typedef). This has never been done mostly because nobody has ever expressed interest in this feature. I will need it myself soon enough. so It should happen at some point soon.

ciao
Alberto
Back to top
montonero
#4 Posted : Monday, March 25, 2013 4:06:44 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/31/2013(UTC)
Posts: 2

Thanks: 0 times
Was thanked: 0 time(s) in 0 post(s)
fagiano wrote:
unfortunately unicode builds right now only work on Windows, I don't think would be too crazy to fix this problem. GCC just missed( or calls differently) a few functions (plus the wchar_t typedef). This has never been done mostly because nobody has ever expressed interest in this feature. I will need it myself soon enough. so It should happen at some point soon.

ciao
Alberto



Hi Alberto,
Just wondering whether you have any plans on implementing the support.
Back to top
fagiano
#5 Posted : Tuesday, March 26, 2013 7:46:12 AM(UTC)
Rank: Advanced Member

Groups: Registered, Administrators
Joined: 6/11/2005(UTC)
Posts: 825

Thanks: 0 times
Was thanked: 36 time(s) in 29 post(s)
Well I will have to do it for my company project at some point. I'm just not sure about the time frame, sometimes this year :) .

The biggest problem is that with gcc wchar_t is 32bits while on win and mac is 16bits so the builds would be somewhat different as the bycode format follows the SQchar size. For my company project I plan to reimplement all the crt functions that we need as 16bits wchar_t because all our codebase is UCS-2. I'm not sure what you are trying to archieve, I think 32bits strings is overkill, I'm not sure why gcc does what it does, probably because on Unix is stuck on UTF-8 and they don't care much about wchar_t. If 32bits is ok for you probably the amount of work is not so much, just figure out the correct match for the windows equivalent( _wcs etc...).
If some Linux internationalization expert has some thought, I'm listening.

ciao
Alberto
Back to top
 |  Edit by user
bonch
#6 Posted : Wednesday, March 27, 2013 2:35:08 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/13/2012(UTC)
Posts: 32

Thanks: 8 times
Was thanked: 1 time(s) in 1 post(s)
Actually, wchar_t on OS X is also UTF-32. I recently made an attempt at implementing SQUNICODE support on OS X, and it turned out to be a little more work than I had anticipated, so I decided to stick with UTF-8 for the time being. NSStrings are UTF-16 internally but are very simple to convert to and from UTF-8.
Back to top
 |  Edit by user
toddkuen
#7 Posted : Sunday, May 12, 2013 12:16:09 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/17/2011(UTC)
Posts: 14
Location: USA

Thanks: 0 times
Was thanked: 2 time(s) in 2 post(s)
I have SQUNICODE working (not fully tested) on OS X (not tested on iOS yet). Basically its not too hard (took a couple of hours).

The steps are

1) Change _SC('X') to something like _SCCHAR('X') - This differentiates char's from strings. This is a simple global edit replace of "_SC('" -> "_SCCHAR('"

2) Define _SC(a) to be u##a - this works because XCode 4 is C++11. Define _SCCHAR(a) as ((unsigned short)(a)).

3) typedef SQChar to be unsigned short

4) Define the following helpers and fix the macros to call these instead of the 'w' ones, e.g., wprintf

extern int SQwprintf(const SQChar * s, ...);
extern unsigned long SQwcstoul( const SQChar * nptr, SQChar ** endptr, int base );
extern double SQwcstod( const SQChar * nptr, SQChar ** endptr );
extern size_t SQwcslen(const SQChar * str);
extern int SQwtoi(const SQChar *);
extern int SQswprintf(SQChar *, const SQChar *, ...);
extern int SQwcscmp(const SQChar *, const SQChar *);
extern int SQvswprintf(SQChar * , const SQChar *, va_list);
extern long long SQ64wcstol(const SQChar *, SQChar **, int);
extern long SQwcstol(const SQChar *, SQChar **, int);
extern SQChar * SQwcsstr(const SQChar * __s1, const SQChar * __s2);

// subset from sqstdio.cpp
extern FILE * _wfopen(const SQChar * path, const SQChar * mode);

// subset from sqstdsystem.cpp
extern SQChar * SQgetenv(const SQChar *);
extern int SQsystem(const SQChar *);
extern SQChar * SQasctime(const struct tm *);
extern int SQremove(const SQChar *);
extern int SQrename (const SQChar *, const SQChar *);

// subset from sqstdstring.cpp
extern SQChar * SQstrchr(const SQChar *, int);
extern int SQsnprintf(SQChar *, size_t, const SQChar *, ...);
extern SQChar * SQstrtok(SQChar *, const SQChar *);

For complex (hard to write from scratch) things like printf I did this by converting the inputs to the mac wprintf (32-bit wchar_t) from SQChar and converting the results back to SQChar. This effectively lets existing wchar_t functions do all the work. (Its not an ideal solution but will work for what I need.)

Other functions I just rewrote or used part of one of my frameworks.

5) There were a hand full of other minor changes. Most important you have to do this in sqbaselib.cpp:

#if defined(__MACOSX__)
if(_ss(v)->_errorfunc) _ss(v)->_errorfunc(v,str);
#else
if(_ss(v)->_errorfunc) _ss(v)->_errorfunc(v,_SC("%s"),str);
#endif // __MACOSX__

The problem is that regular printf and wprintf (and friends) cannot deal with '%s' referring to a SQChar * as shorts. Since the base_* functions have to translate "outside" of Squirrel to UTF-8 or something this gets tricky. The hack above works for now.

Contact me at "todd" at "synthodeon" dot "com" if you need more info. I didn't post my implementation of the helpers because it uses some of my frameworks - which might confuse people - but I will make them available if anyone wants them.
Back to top
WWW
 |  Edit by user
1 user thanked toddkuen for this useful post.
bonch on 5/13/2013(UTC)
toddkuen
#8 Posted : Sunday, May 12, 2013 1:53:10 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/17/2011(UTC)
Posts: 14
Location: USA

Thanks: 0 times
Was thanked: 2 time(s) in 2 post(s)
On point #5 - the real problem is that '%s' is really ambiguous and defined by the context Squirrel is compiled in.

Printf is the wrong thing to use because of this. You need a descriptive string and corresponding output mechanism that removes this ambiguity.

The final bit of code I need is something that scans through printf strings for '%s' and friends and handles the corresponding SQChar * pointer in an appropriate manner.

Raise_Error does this all over the place so the solution needs to handle arbitrary printf strings I guess...
Back to top
WWW
toddkuen
#9 Posted : Monday, May 13, 2013 1:54:22 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/17/2011(UTC)
Posts: 14
Location: USA

Thanks: 0 times
Was thanked: 2 time(s) in 2 post(s)
I ended up using this code for a printf replacement:

Code:

int _SCprintf(const SQChar * format, ...)
{
  int cnt, pcnt, j;
  const SQChar * p, * ep;
  wchar_t * fp, fType;
  wchar_t formBuf[128], pntOut[SQMAX_PRINT_LENGTH];
  va_list args;
 
  va_start (args, format);
  cnt = 0;
  p = format;
  while (*p != 0)
  {
    if (*p == '%')
    {
      if (*(p + 1) == '%')
      {
        p++;
        goto out1;
      }
      else
      {
        // Extract the format
        //
        formBuf[0] = *p;
        ep = p + 1;
        fp = &formBuf[1];
        while (!endFormatChar(*ep))
        {
          *fp++ = *ep++;
        }
        fType = *ep;
        *fp = *ep;
        *(fp + 1) = 0;
       
        if (fType == 's')
        {
          const SQChar * sp;
         
          *fp = 'l';
          *(fp + 1) = 's';
          *(fp + 2) = 0;
         
          sp = va_arg(args, const SQChar *);
         
          pcnt = swprintf(pntOut, SQMAX_PRINT_LENGTH, formBuf, toWCharT(sp));
        }
        else
        {
          pcnt = vswprintf(pntOut, SQMAX_PRINT_LENGTH, formBuf, args);
        }
       
        for (j = 0; j < pcnt; j++)
        {
          _SCprintfOutC(stdout, pntOut[j]);
        }
        cnt += pcnt;
       
        p = ep + 1;
      }
    }
    else
    {
    out1:
      _SCprintfOutC(stdout, *p++);
      cnt++;
    }
  }
 
  va_end (args);
 
  return cnt;
}


where _SCprintfOutC is defined as

Code:
void _SCprintfOutC(FILE * f, int aChar)
{
#if defined(SQPRINTFTOWIDE)
  fputwc(aChar, f);
#endif // SQPRINTFTOWIDE
#if defined(SQPRINTFTOUTF8)
  {
    SQChar buf[2];
   
    buf[0] = 0xFFFF & aChar;
    buf[1] = 0;
    fputs(ZMDICore::nsToUTF8Chars(buf).c_str(), f);
  }
#endif // SQPRINTFTOUTF8
#if defined(SQPRINTFTOASCII)
  fputc(0x7F & aChar, f);
#endif // SQPRINTFTOASCII
#if defined(SQPRINTFTOSQCHAR)
  // This should match whatever happens in Windows (if anything...)
  // for writing out wide characters.
  //
  fputc(0xFF & aChar, f);
  fputc(0xFF & (aChar >> 8), f);
#endif // SQPRINTFTOSQCHAR
}


You can see the issues. Its unclear what you want to print with printf on OS X in SQChar as int-16: UTF-8, ASCII, UNICODE?

My solution is basically to on-the-fly reinterpret '%s' - while this works its probably not the best long term solution.
Back to top
WWW
 |  Edit by user
1 user thanked toddkuen for this useful post.
bonch on 5/13/2013(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

Clean Slate theme by Jaben Cargman (Tiny Gecko)
Powered by YAF 1.9.4 | YAF © 2003-2010, Yet Another Forum.NET
This page was generated in 0.178 seconds.