eo - by default on 64bit only use 47 bits because of luajit

luajit wants to check the upper 17 bits of pointers and assume they
are all 0 if on a 64bit architecture. it will panic and barf if they
are used. we have been using them in our eoid's for a long time.

my take on this is that this is unportable. assuming how many bits are
or are not valid in an opaque pointer (void *) that is simply passed
to you and you cannot dereference or use (no size information even for
what amount of data it points to etc.), is just wrong. it's not
portable and it's trying to be too smart and creating such issues. my
take is that luajit needs a fix for this in the longer term. but for
now let's have a 47 bit mode and go with that. it does mean i have to
drop our generation counter to 10 bits on 64bit (from 27 bits) which
increases likelihood of eoid re-use being falsely detected as valid
(before on 64bit it was 1 in 130 million or so chance, with this 47
bit change it's 1 in 1000. but to be fair on 32bit it's 7 bits for gen
count so 1 in 127 ... so still more than 10x "safer" than on 32bit...
but still...). the relevant check in luajit is:

  (((uint64_t)(p) >> 47) ? (lj_err_msg(L, LJ_ERR_BADLU), NULL) : (p))

it ONLY does this on 64bit. on 32bit pointers are not checked for
validity at all.

as an aside, armv8.2 seemingly will bring 52bit addresses so luajit is
going to fall over flat on a newer rev of armv8. it may be x86 also
uses more bits. last i knew it was 48bits so the 47 bit check luajit
does i think is even wrong for x86-64. to be detailed i read:

amd64 arch == 48 bits (so luajit is wrong). even better In addition,
the AMD specification requires that the most significant 16 bits of
any virtual address, bits 48 through 63, must be copies of bit 47 (in
a manner akin to sign extension). so if the upper bit of 48 is set
THEN all the 16 upper bits must be 1... breaking luajit, even if it
were 47bit and this rule applied. I read the architecture allows for
up to 52bits of actual addresses so architecture-wise this is even
wrong...

So I smell a core bug here in luajit. Certainly in the number of bits
it insists must be 0 (the upper 17 bits where on amd64/x86-64 it
should be the upper 16 bits... and even then these may NOT be 0 if bit
47 (the upper bit of the lower 48 is 1).... so the whole check is
invalid... :(

at least the above is at a theoretical level. i believe that the
addresses divide the 48 bits into 2 chunks (thus 47)... but at the
PHYSICAL level with no mmu and virtual memory. arm64 has this:

https://www.kernel.org/doc/Documentation/arm64/memory.txt

note in all cases the 2nd chunk of memory has at leats some upper bits
of physical addresses beign 1 ... which makes luajit invalid tyo use
without virtual memory remapping these away from high bits.

@fix

1 files changed, 30 insertions, 9 deletions

diff --git a/src/lib/eo/eo_ptr_indirection.x b/src/lib/eo/eo_ptr_indirection.x
index d9bd863cc7..f4311e1c3a 100644
--- a/src/lib/eo/eo_ptr_indirection.x
+++ b/src/lib/eo/eo_ptr_indirection.x
@@ -57,6 +57,12 @@
  * it to the fifo.
  */
 
+// enable this to test and use all 64bits of a pointer, otherwise limit to
+// 47 bits because of luajit. it wants to check if any bits in the upper 17 are
+// set for a sanity check for lightuserdata ... basically it does this:
+// #define checklightudptr(L, p) (((uint64_t)(p) >> 47) ? (lj_err_msg(L, LJ_ERR_BADLU), NULL) : (p))
+//#define EO_FULL64BIT 1
+
 #if SIZEOF_UINTPTR_T == 4
 /* 32 bits */
 # define BITS_MID_TABLE_ID        5
@@ -71,18 +77,33 @@
 typedef int16_t Table_Index;
 typedef uint16_t Generation_Counter;
 #else
+# ifndef EO_FULL64BIT
+/* 47 bits */
+#  define BITS_MID_TABLE_ID       11
+#  define BITS_TABLE_ID           11
+#  define BITS_ENTRY_ID           11
+#  define BITS_GENERATION_COUNTER 10
+#  define BITS_DOMAIN              2
+#  define BITS_CLASS               1
+#  define REF_TAG_SHIFT           46
+#  define DROPPED_TABLES           2
+#  define DROPPED_ENTRIES          3
+typedef int16_t Table_Index;
+typedef uint16_t Generation_Counter;
+# else
 /* 64 bits */
-# define BITS_MID_TABLE_ID       11
-# define BITS_TABLE_ID           11
-# define BITS_ENTRY_ID           11
-# define BITS_GENERATION_COUNTER 27
-# define BITS_DOMAIN              2
-# define BITS_CLASS               1
-# define REF_TAG_SHIFT           63
-# define DROPPED_TABLES           2
-# define DROPPED_ENTRIES          3
+#  define BITS_MID_TABLE_ID       11
+#  define BITS_TABLE_ID           11
+#  define BITS_ENTRY_ID           11
+#  define BITS_GENERATION_COUNTER 27
+#  define BITS_DOMAIN              2
+#  define BITS_CLASS               1
+#  define REF_TAG_SHIFT           63
+#  define DROPPED_TABLES           2
+#  define DROPPED_ENTRIES          3
 typedef int16_t Table_Index;
 typedef uint32_t Generation_Counter;
+# endif
 #endif
 
 /* Shifts macros to manipulate the Eo id */