From 858e9961e2d9291ef47ee3a9e23971ac429121b3 Mon Sep 17 00:00:00 2001 From: Wouter van Oortmerssen Date: Wed, 10 Sep 2014 16:38:37 -0700 Subject: [PATCH] Added accessor in Java to get vectors as ByteBuffers. Also cleaned up ByteBuffer usage in general: ByteBuffer.position now universally indicates the start of a ByteBuffer. Change-Id: Ic4bfb98f9df9501b8fc82de2c45db7d7311135ac Tested: on Linux. --- docs/html/md__java_usage.html | 7 ++++--- docs/source/JavaUsage.md | 27 ++++++++++++++++++--------- java/flatbuffers/FlatBufferBuilder.java | 16 ++++++++++++---- java/flatbuffers/Table.java | 28 ++++++++++++++++++++++++---- src/idl_gen_java.cpp | 17 +++++++++++++---- tests/JavaTest.java | 21 ++++++++++++++------- tests/MyGame/Example/Monster.java | 10 ++++++++-- 7 files changed, 93 insertions(+), 33 deletions(-) diff --git a/docs/html/md__java_usage.html b/docs/html/md__java_usage.html index b9b9ee0..b3d353b 100644 --- a/docs/html/md__java_usage.html +++ b/docs/html/md__java_usage.html @@ -60,10 +60,11 @@ Monster monster = Monster.getRootAsMonster(bb); Vec3 pos = monster.pos();

Note that whenever you access a new object like in the pos example above, a new temporary accessor object gets created. If your code is very performance sensitive (you iterate through a lot of objects), there's a second pos() method to which you can pass a Vec3 object you've already created. This allows you to reuse it across many calls and reduce the amount of object allocation (and thus garbage collection) your program does.

Java does not support unsigned scalars. This means that any unsigned types you use in your schema will actually be represented as a signed value. This means all bits are still present, but may represent a negative value when used. For example, to read a byte b as an unsigned number, you can do: (short)(b & 0xFF)

-

Sadly the string accessors currently always create a new string when accessed, since FlatBuffer's UTF-8 strings can't be read in-place by Java.

+

The default string accessor (e.g. monster.name()) currently always create a new Java String when accessed, since FlatBuffer's UTF-8 strings can't be used in-place by String. Alternatively, use monster.nameAsByteBuffer() which returns a ByteBuffer referring to the UTF-8 data in the original ByteBuffer, which is much more efficient. The ByteBuffer's position points to the first character, and its limit to just after the last.

Vector access is also a bit different from C++: you pass an extra index to the vector field accessor. Then a second method with the same name suffixed by Length let's you know the number of elements you can access: 

for (int i = 0; i < monster.inventoryLength(); i++)
     monster.inventory(i); // do something here
-

If you specified a file_indentifier in the schema, you can query if the buffer is of the desired type before accessing it using: 

if (Monster.MonsterBufferHasIdentifier(bb, start)) ...
+

Alternatively, much like strings, you can use monster.inventoryAsByteBuffer() to get a ByteBuffer referring to the whole vector. Use ByteBuffer methods like asFloatBuffer to get specific views if needed.

+

If you specified a file_indentifier in the schema, you can query if the buffer is of the desired type before accessing it using: 

if (Monster.MonsterBufferHasIdentifier(bb)) ...

Buffer construction in Java

You can also construct these buffers in Java using the static methods found in the generated code, and the FlatBufferBuilder class: 

FlatBufferBuilder fbb = new FlatBufferBuilder();

Create strings: 

int str = fbb.createString("MyMonster");
@@ -86,7 +87,7 @@ int inv = fbb.endVector();

You can use the generated method startInventoryVector to conveniently call startVector with the right element size. You pass the number of elements you want to write. You write the elements backwards since the buffer is being constructed back to front.

There are add functions for all the scalar types. You use addOffset for any previously constructed objects (such as other tables, strings, vectors). For structs, you use the appropriate create function in-line, as shown above in the Monster example.

To finish the buffer, call: 

Monster.finishMonsterBuffer(fbb, mon);
-

The buffer is now ready to be transmitted. It is contained in the ByteBuffer which you can obtain from fbb.dataBuffer(). Importantly, the valid data does not start from offset 0 in this buffer, but from fbb.dataStart() (this is because the data was built backwards in memory). It ends at fbb,capacity().

+

The buffer is now ready to be transmitted. It is contained in the ByteBuffer which you can obtain from fbb.dataBuffer(). Importantly, the valid data does not start from offset 0 in this buffer, but from fbb.dataBuffer().position() (this is because the data was built backwards in memory). It ends at fbb.capacity().

Text Parsing

There currently is no support for parsing text (Schema's and JSON) directly from Java, though you could use the C++ parser through JNI. Please see the C++ documentation for more on text parsing.

diff --git a/docs/source/JavaUsage.md b/docs/source/JavaUsage.md index 15bd816..0c1e2d5 100755 --- a/docs/source/JavaUsage.md +++ b/docs/source/JavaUsage.md @@ -1,7 +1,7 @@ # Use in Java -FlatBuffers supports reading and writing binary FlatBuffers in Java. Generate code -for Java with the `-j` option to `flatc`. +FlatBuffers supports reading and writing binary FlatBuffers in Java. Generate +code for Java with the `-j` option to `flatc`. See `javaTest.java` for an example. Essentially, you read a FlatBuffer binary file into a `byte[]`, which you then turn into a `ByteBuffer`, which you pass to @@ -19,8 +19,8 @@ Note that whenever you access a new object like in the `pos` example above, a new temporary accessor object gets created. If your code is very performance sensitive (you iterate through a lot of objects), there's a second `pos()` method to which you can pass a `Vec3` object you've already created. This allows -you to reuse it across many calls and reduce the amount of object allocation (and -thus garbage collection) your program does. +you to reuse it across many calls and reduce the amount of object allocation +(and thus garbage collection) your program does. Java does not support unsigned scalars. This means that any unsigned types you use in your schema will actually be represented as a signed value. This means @@ -28,8 +28,12 @@ all bits are still present, but may represent a negative value when used. For example, to read a `byte b` as an unsigned number, you can do: `(short)(b & 0xFF)` -Sadly the string accessors currently always create a new string when accessed, -since FlatBuffer's UTF-8 strings can't be read in-place by Java. +The default string accessor (e.g. `monster.name()`) currently always create +a new Java `String` when accessed, since FlatBuffer's UTF-8 strings can't be +used in-place by `String`. Alternatively, use `monster.nameAsByteBuffer()` +which returns a `ByteBuffer` referring to the UTF-8 data in the original +`ByteBuffer`, which is much more efficient. The `ByteBuffer`'s `position` +points to the first character, and its `limit` to just after the last. Vector access is also a bit different from C++: you pass an extra index to the vector field accessor. Then a second method with the same name @@ -38,10 +42,14 @@ suffixed by `Length` let's you know the number of elements you can access: for (int i = 0; i < monster.inventoryLength(); i++) monster.inventory(i); // do something here +Alternatively, much like strings, you can use `monster.inventoryAsByteBuffer()` +to get a `ByteBuffer` referring to the whole vector. Use `ByteBuffer` methods +like `asFloatBuffer` to get specific views if needed. + If you specified a file_indentifier in the schema, you can query if the buffer is of the desired type before accessing it using: - if (Monster.MonsterBufferHasIdentifier(bb, start)) ... + if (Monster.MonsterBufferHasIdentifier(bb)) ... ## Buffer construction in Java @@ -105,8 +113,9 @@ To finish the buffer, call: The buffer is now ready to be transmitted. It is contained in the `ByteBuffer` which you can obtain from `fbb.dataBuffer()`. Importantly, the valid data does -not start from offset 0 in this buffer, but from `fbb.dataStart()` (this is -because the data was built backwards in memory). It ends at `fbb,capacity()`. +not start from offset 0 in this buffer, but from `fbb.dataBuffer().position()` +(this is because the data was built backwards in memory). +It ends at `fbb.capacity()`. ## Text Parsing diff --git a/java/flatbuffers/FlatBufferBuilder.java b/java/flatbuffers/FlatBufferBuilder.java index 4623661..c1fa771 100755 --- a/java/flatbuffers/FlatBufferBuilder.java +++ b/java/flatbuffers/FlatBufferBuilder.java @@ -245,6 +245,7 @@ public class FlatBufferBuilder { public void finish(int root_table) { prep(minalign, SIZEOF_INT); addOffset(root_table); + bb.position(space); } public void finish(int root_table, String file_identifier) { @@ -255,13 +256,20 @@ public class FlatBufferBuilder { for (int i = FILE_IDENTIFIER_LENGTH - 1; i >= 0; i--) { addByte((byte)file_identifier.charAt(i)); } - addOffset(root_table); + finish(root_table); } + // Get the ByteBuffer representing the FlatBuffer. Only call this after you've + // called finish(). The actual data starts at the ByteBuffer's current position, + // not necessarily at 0. public ByteBuffer dataBuffer() { return bb; } - // The FlatBuffer data doesn't start at offset 0 in the ByteBuffer: - public int dataStart() { + // The FlatBuffer data doesn't start at offset 0 in the ByteBuffer, + // but now the ByteBuffer's position is set to that location upon + // finish(). This method should not be needed anymore, but is left + // here as private for the moment to document this API change. + // It will be removed in the future. + private int dataStart() { return space; } @@ -273,7 +281,7 @@ public class FlatBufferBuilder { } // Utility function for copying a byte array that starts at 0. - public byte[] sizedByteArray(){ + public byte[] sizedByteArray() { return sizedByteArray(space, bb.capacity() - space); } } diff --git a/java/flatbuffers/Table.java b/java/flatbuffers/Table.java index 4daafff..d48a325 100755 --- a/java/flatbuffers/Table.java +++ b/java/flatbuffers/Table.java @@ -38,6 +38,10 @@ public class Table { } // Create a java String from UTF-8 data stored inside the flatbuffer. + // This allocates a new string and converts to wide chars upon each access, + // which is not very efficient. Instead, each FlatBuffer string also comes with an + // accessor based on __vector_as_bytebuffer below, which is much more efficient, + // assuming your Java program can handle UTF-8 data directly. protected String __string(int offset) { offset += bb.getInt(offset); if (bb.hasArray()) { @@ -45,10 +49,11 @@ public class Table { } else { // We can't access .array(), since the ByteBuffer is read-only. // We're forced to make an extra copy: - bb.position(offset + SIZEOF_INT); byte[] copy = new byte[bb.getInt(offset)]; + int old_pos = bb.position(); + bb.position(offset + SIZEOF_INT); bb.get(copy); - bb.position(0); + bb.position(old_pos); return new String(copy, 0, copy.length, Charset.forName("UTF-8")); } } @@ -66,6 +71,21 @@ public class Table { return offset + bb.getInt(offset) + SIZEOF_INT; // data starts after the length } + // Get a whole vector as a ByteBuffer. This is efficient, since it only allocates a new + // bytebuffer object, but does not actually copy the data, it still refers to the same + // bytes as the original ByteBuffer. + // Also useful with nested FlatBuffers etc. + protected ByteBuffer __vector_as_bytebuffer(int vector_offset, int elem_size) { + int o = __offset(vector_offset); + if (o == 0) return null; + int old_pos = bb.position(); + bb.position(__vector(o)); + ByteBuffer nbb = bb.slice(); + bb.position(old_pos); + nbb.limit(__vector_len(o) * elem_size); + return nbb; + } + // Initialize any Table-derived type to point to the union at the given offset. protected Table __union(Table t, int offset) { offset += bb_pos; @@ -74,12 +94,12 @@ public class Table { return t; } - protected static boolean __has_identifier(ByteBuffer bb, int offset, String ident) { + protected static boolean __has_identifier(ByteBuffer bb, String ident) { if (ident.length() != FILE_IDENTIFIER_LENGTH) throw new AssertionError("FlatBuffers: file identifier must be length " + FILE_IDENTIFIER_LENGTH); for (int i = 0; i < FILE_IDENTIFIER_LENGTH; i++) { - if (ident.charAt(i) != (char)bb.get(offset + SIZEOF_INT + i)) return false; + if (ident.charAt(i) != (char)bb.get(bb.position() + SIZEOF_INT + i)) return false; } return true; } diff --git a/src/idl_gen_java.cpp b/src/idl_gen_java.cpp index 8ee4200..91847b5 100755 --- a/src/idl_gen_java.cpp +++ b/src/idl_gen_java.cpp @@ -174,16 +174,16 @@ static void GenStruct(const Parser &parser, StructDef &struct_def, // of a FlatBuffer code += " public static " + struct_def.name + " getRootAs"; code += struct_def.name; - code += "(ByteBuffer _bb, int offset) { "; + code += "(ByteBuffer _bb) { "; code += "_bb.order(ByteOrder.LITTLE_ENDIAN); "; code += "return (new " + struct_def.name; - code += "()).__init(_bb.getInt(offset) + offset, _bb); }\n"; + code += "()).__init(_bb.getInt(_bb.position()) + _bb.position(), _bb); }\n"; if (parser.root_struct_def == &struct_def) { if (parser.file_identifier_.length()) { // Check if a buffer has the identifier. code += " public static boolean " + struct_def.name; - code += "BufferHasIdentifier(ByteBuffer _bb, int offset) { return "; - code += "__has_identifier(_bb, offset, \"" + parser.file_identifier_; + code += "BufferHasIdentifier(ByteBuffer _bb) { return "; + code += "__has_identifier(_bb, \"" + parser.file_identifier_; code += "\"); }\n"; } } @@ -285,6 +285,15 @@ static void GenStruct(const Parser &parser, StructDef &struct_def, code += offset_prefix; code += "__vector_len(o) : 0; }\n"; } + if (field.value.type.base_type == BASE_TYPE_VECTOR || + field.value.type.base_type == BASE_TYPE_STRING) { + code += " public ByteBuffer " + MakeCamel(field.name, false); + code += "AsByteBuffer() { return __vector_as_bytebuffer("; + code += NumToString(field.value.offset) + ", "; + code += NumToString(field.value.type.base_type == BASE_TYPE_STRING ? 1 : + InlineSize(field.value.type.VectorType())); + code += "); }\n"; + } } code += "\n"; if (struct_def.fixed) { diff --git a/tests/JavaTest.java b/tests/JavaTest.java index 04646c5..74d68db 100755 --- a/tests/JavaTest.java +++ b/tests/JavaTest.java @@ -41,7 +41,7 @@ class JavaTest { // Now test it: ByteBuffer bb = ByteBuffer.wrap(data); - TestBuffer(bb, 0); + TestBuffer(bb); // Second, let's create a FlatBuffer from scratch in Java, and test it also. // We use an initial size of 1 to exercise the reallocation algorithm, @@ -95,7 +95,7 @@ class JavaTest { try { DataOutputStream os = new DataOutputStream(new FileOutputStream( "monsterdata_java_wire.bin")); - os.write(fbb.dataBuffer().array(), fbb.dataStart(), fbb.offset()); + os.write(fbb.dataBuffer().array(), fbb.dataBuffer().position(), fbb.offset()); os.close(); } catch(java.io.IOException e) { System.out.println("FlatBuffers test: couldn't write file"); @@ -103,20 +103,20 @@ class JavaTest { } // Test it: - TestBuffer(fbb.dataBuffer(), fbb.dataStart()); + TestBuffer(fbb.dataBuffer()); // Make sure it also works with read only ByteBuffers. This is slower, // since creating strings incurs an additional copy // (see Table.__string). - TestBuffer(fbb.dataBuffer().asReadOnlyBuffer(), fbb.dataStart()); + TestBuffer(fbb.dataBuffer().asReadOnlyBuffer()); System.out.println("FlatBuffers test: completed successfully"); } - static void TestBuffer(ByteBuffer bb, int start) { - TestEq(Monster.MonsterBufferHasIdentifier(bb, start), true); + static void TestBuffer(ByteBuffer bb) { + TestEq(Monster.MonsterBufferHasIdentifier(bb), true); - Monster monster = Monster.getRootAsMonster(bb, start); + Monster monster = Monster.getRootAsMonster(bb); TestEq(monster.hp(), (short)80); TestEq(monster.mana(), (short)150); // default @@ -145,6 +145,13 @@ class JavaTest { invsum += monster.inventory(i); TestEq(invsum, 10); + // Alternative way of accessing a vector: + ByteBuffer ibb = monster.inventoryAsByteBuffer(); + invsum = 0; + while (ibb.position() < ibb.limit()) + invsum += ibb.get(); + TestEq(invsum, 10); + Test test_0 = monster.test4(0); Test test_1 = monster.test4(1); TestEq(monster.test4Length(), 2); diff --git a/tests/MyGame/Example/Monster.java b/tests/MyGame/Example/Monster.java index 76f9476..fa7bc21 100755 --- a/tests/MyGame/Example/Monster.java +++ b/tests/MyGame/Example/Monster.java @@ -8,8 +8,8 @@ import java.util.*; import flatbuffers.*; public class Monster extends Table { - public static Monster getRootAsMonster(ByteBuffer _bb, int offset) { _bb.order(ByteOrder.LITTLE_ENDIAN); return (new Monster()).__init(_bb.getInt(offset) + offset, _bb); } - public static boolean MonsterBufferHasIdentifier(ByteBuffer _bb, int offset) { return __has_identifier(_bb, offset, "MONS"); } + public static Monster getRootAsMonster(ByteBuffer _bb) { _bb.order(ByteOrder.LITTLE_ENDIAN); return (new Monster()).__init(_bb.getInt(_bb.position()) + _bb.position(), _bb); } + public static boolean MonsterBufferHasIdentifier(ByteBuffer _bb) { return __has_identifier(_bb, "MONS"); } public Monster __init(int _i, ByteBuffer _bb) { bb_pos = _i; bb = _bb; return this; } public Vec3 pos() { return pos(new Vec3()); } @@ -17,24 +17,30 @@ public class Monster extends Table { public short mana() { int o = __offset(6); return o != 0 ? bb.getShort(o + bb_pos) : 150; } public short hp() { int o = __offset(8); return o != 0 ? bb.getShort(o + bb_pos) : 100; } public String name() { int o = __offset(10); return o != 0 ? __string(o + bb_pos) : null; } + public ByteBuffer nameAsByteBuffer() { return __vector_as_bytebuffer(10, 1); } public byte inventory(int j) { int o = __offset(14); return o != 0 ? bb.get(__vector(o) + j * 1) : 0; } public int inventoryLength() { int o = __offset(14); return o != 0 ? __vector_len(o) : 0; } + public ByteBuffer inventoryAsByteBuffer() { return __vector_as_bytebuffer(14, 1); } public byte color() { int o = __offset(16); return o != 0 ? bb.get(o + bb_pos) : 8; } public byte testType() { int o = __offset(18); return o != 0 ? bb.get(o + bb_pos) : 0; } public Table test(Table obj) { int o = __offset(20); return o != 0 ? __union(obj, o) : null; } public Test test4(int j) { return test4(new Test(), j); } public Test test4(Test obj, int j) { int o = __offset(22); return o != 0 ? obj.__init(__vector(o) + j * 4, bb) : null; } public int test4Length() { int o = __offset(22); return o != 0 ? __vector_len(o) : 0; } + public ByteBuffer test4AsByteBuffer() { return __vector_as_bytebuffer(22, 4); } public String testarrayofstring(int j) { int o = __offset(24); return o != 0 ? __string(__vector(o) + j * 4) : null; } public int testarrayofstringLength() { int o = __offset(24); return o != 0 ? __vector_len(o) : 0; } + public ByteBuffer testarrayofstringAsByteBuffer() { return __vector_as_bytebuffer(24, 4); } /// an example documentation comment: this will end up in the generated code multiline too public Monster testarrayoftables(int j) { return testarrayoftables(new Monster(), j); } public Monster testarrayoftables(Monster obj, int j) { int o = __offset(26); return o != 0 ? obj.__init(__indirect(__vector(o) + j * 4), bb) : null; } public int testarrayoftablesLength() { int o = __offset(26); return o != 0 ? __vector_len(o) : 0; } + public ByteBuffer testarrayoftablesAsByteBuffer() { return __vector_as_bytebuffer(26, 4); } public Monster enemy() { return enemy(new Monster()); } public Monster enemy(Monster obj) { int o = __offset(28); return o != 0 ? obj.__init(__indirect(o + bb_pos), bb) : null; } public byte testnestedflatbuffer(int j) { int o = __offset(30); return o != 0 ? bb.get(__vector(o) + j * 1) : 0; } public int testnestedflatbufferLength() { int o = __offset(30); return o != 0 ? __vector_len(o) : 0; } + public ByteBuffer testnestedflatbufferAsByteBuffer() { return __vector_as_bytebuffer(30, 1); } public Monster testempty() { return testempty(new Monster()); } public Monster testempty(Monster obj) { int o = __offset(32); return o != 0 ? obj.__init(__indirect(o + bb_pos), bb) : null; } -- 2.7.4