From 0eac15c784e8071f2b517e8f8420380a8f3326d6 Mon Sep 17 00:00:00 2001 From: Wouter van Oortmerssen Date: Mon, 6 Oct 2014 10:43:02 -0700 Subject: [PATCH] Added fenced code blocks to the C++/Java/Go docs for syntax highlighting. Change-Id: I504915c6b5367e8c05dc056463158b8420ad8c5e Tested: on Linux. --- docs/html/md__cpp_usage.html | 86 ++++++++++++++++++++++++------------------- docs/html/md__go_usage.html | 75 ++++++++++++++++++++----------------- docs/html/md__java_usage.html | 68 +++++++++++++++++++--------------- docs/source/CppUsage.md | 26 +++++++++++++ docs/source/GoUsage.md | 14 +++++++ docs/source/JavaUsage.md | 20 ++++++++++ 6 files changed, 189 insertions(+), 100 deletions(-) diff --git a/docs/html/md__cpp_usage.html b/docs/html/md__cpp_usage.html index 922a33c..746bbb1 100644 --- a/docs/html/md__cpp_usage.html +++ b/docs/html/md__cpp_usage.html @@ -55,42 +55,52 @@ $(document).ready(function(){initNavTree('md__cpp_usage.html','');});

Assuming you have written a schema using the above language in say mygame.fbs (FlatBuffer Schema, though the extension doesn't matter), you've generated a C++ header called mygame_generated.h using the compiler (e.g. flatc -c mygame.fbs), you can now start using this in your program by including the header. As noted, this header relies on flatbuffers/flatbuffers.h, which should be in your include path.

Writing in C++

-

To start creating a buffer, create an instance of FlatBufferBuilder which will contain the buffer as it grows: 

FlatBufferBuilder fbb;
-

Before we serialize a Monster, we need to first serialize any objects that are contained there-in, i.e. we serialize the data tree using depth first, pre-order traversal. This is generally easy to do on any tree structures. For example: 

auto name = fbb.CreateString("MyMonster");
-
-unsigned char inv[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
-auto inventory = fbb.CreateVector(inv, 10);
-

CreateString and CreateVector serialize these two built-in datatypes, and return offsets into the serialized data indicating where they are stored, such that Monster below can refer to them.

+

To start creating a buffer, create an instance of FlatBufferBuilder which will contain the buffer as it grows:

+
FlatBufferBuilder fbb;
+

Before we serialize a Monster, we need to first serialize any objects that are contained there-in, i.e. we serialize the data tree using depth first, pre-order traversal. This is generally easy to do on any tree structures. For example:

+
auto name = fbb.CreateString("MyMonster");
+
+
unsigned char inv[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
auto inventory = fbb.CreateVector(inv, 10);
+

CreateString and CreateVector serialize these two built-in datatypes, and return offsets into the serialized data indicating where they are stored, such that Monster below can refer to them.

CreateString can also take an std::string, or a const char * with an explicit length, and is suitable for holding UTF-8 and binary data if needed.

-

CreateVector can also take an std::vector. The offset it returns is typed, i.e. can only be used to set fields of the correct type below. To create a vector of struct objects (which will be stored as contiguous memory in the buffer, use CreateVectorOfStructs instead. 

Vec3 vec(1, 2, 3);
-

Vec3 is the first example of code from our generated header. Structs (unlike tables) translate to simple structs in C++, so we can construct them in a familiar way.

-

We have now serialized the non-scalar components of of the monster example, so we could create the monster something like this: 

auto mloc = CreateMonster(fbb, &vec, 150, 80, name, inventory, Color_Red, 0, Any_NONE);
-

Note that we're passing 150 for the mana field, which happens to be the default value: this means the field will not actually be written to the buffer, since we'll get that value anyway when we query it. This is a nice space savings, since it is very common for fields to be at their default. It means we also don't need to be scared to add fields only used in a minority of cases, since they won't bloat up the buffer sizes if they're not actually used.

+

CreateVector can also take an std::vector. The offset it returns is typed, i.e. can only be used to set fields of the correct type below. To create a vector of struct objects (which will be stored as contiguous memory in the buffer, use CreateVectorOfStructs instead.

+
Vec3 vec(1, 2, 3);
+

Vec3 is the first example of code from our generated header. Structs (unlike tables) translate to simple structs in C++, so we can construct them in a familiar way.

+

We have now serialized the non-scalar components of of the monster example, so we could create the monster something like this:

+
auto mloc = CreateMonster(fbb, &vec, 150, 80, name, inventory, Color_Red, 0, Any_NONE);
+

Note that we're passing 150 for the mana field, which happens to be the default value: this means the field will not actually be written to the buffer, since we'll get that value anyway when we query it. This is a nice space savings, since it is very common for fields to be at their default. It means we also don't need to be scared to add fields only used in a minority of cases, since they won't bloat up the buffer sizes if they're not actually used.

We do something similarly for the union field test by specifying a 0 offset and the NONE enum value (part of every union) to indicate we don't actually want to write this field. You can use 0 also as a default for other non-scalar types, such as strings, vectors and tables.

-

Tables (like Monster) give you full flexibility on what fields you write (unlike Vec3, which always has all fields set because it is a struct). If you want even more control over this (i.e. skip fields even when they are not default), instead of the convenient CreateMonster call we can also build the object field-by-field manually: 

MonsterBuilder mb(fbb);
-mb.add_pos(&vec);
-mb.add_hp(80);
-mb.add_name(name);
-mb.add_inventory(inventory);
-auto mloc = mb.Finish();
-

We start with a temporary helper class MonsterBuilder (which is defined in our generated code also), then call the various add_ methods to set fields, and Finish to complete the object. This is pretty much the same code as you find inside CreateMonster, except we're leaving out a few fields. Fields may also be added in any order, though orderings with fields of the same size adjacent to each other most efficient in size, due to alignment. You should not nest these Builder classes (serialize your data in pre-order).

-

Regardless of whether you used CreateMonster or MonsterBuilder, you now have an offset to the root of your data, and you can finish the buffer using: 

FinishMonsterBuffer(fbb, mloc);
-

The buffer is now ready to be stored somewhere, sent over the network, be compressed, or whatever you'd like to do with it. You can access the start of the buffer with fbb.GetBufferPointer(), and it's size from fbb.GetSize().

+

Tables (like Monster) give you full flexibility on what fields you write (unlike Vec3, which always has all fields set because it is a struct). If you want even more control over this (i.e. skip fields even when they are not default), instead of the convenient CreateMonster call we can also build the object field-by-field manually:

+
MonsterBuilder mb(fbb);
+
mb.add_pos(&vec);
+
mb.add_hp(80);
+
mb.add_name(name);
+
mb.add_inventory(inventory);
+
auto mloc = mb.Finish();
+

We start with a temporary helper class MonsterBuilder (which is defined in our generated code also), then call the various add_ methods to set fields, and Finish to complete the object. This is pretty much the same code as you find inside CreateMonster, except we're leaving out a few fields. Fields may also be added in any order, though orderings with fields of the same size adjacent to each other most efficient in size, due to alignment. You should not nest these Builder classes (serialize your data in pre-order).

+

Regardless of whether you used CreateMonster or MonsterBuilder, you now have an offset to the root of your data, and you can finish the buffer using:

+
FinishMonsterBuffer(fbb, mloc);
+

The buffer is now ready to be stored somewhere, sent over the network, be compressed, or whatever you'd like to do with it. You can access the start of the buffer with fbb.GetBufferPointer(), and it's size from fbb.GetSize().

samples/sample_binary.cpp is a complete code sample similar to the code above, that also includes the reading code below.

Reading in C++

-

If you've received a buffer from somewhere (disk, network, etc.) you can directly start traversing it using: 

auto monster = GetMonster(buffer_pointer);
-

monster is of type Monster *, and points to somewhere inside your buffer. If you look in your generated header, you'll see it has convenient accessors for all fields, e.g. 

assert(monster->hp() == 80);
-assert(monster->mana() == 150);  // default
-assert(strcmp(monster->name()->c_str(), "MyMonster") == 0);
-

These should all be true. Note that we never stored a mana value, so it will return the default.

-

To access sub-objects, in this case the Vec3: 

auto pos = monster->pos();
-assert(pos);
-assert(pos->z() == 3);
-

If we had not set the pos field during serialization, it would be NULL.

-

Similarly, we can access elements of the inventory array: 

auto inv = monster->inventory();
-assert(inv);
-assert(inv->Get(9) == 9);
-

Direct memory access

+

If you've received a buffer from somewhere (disk, network, etc.) you can directly start traversing it using:

+
auto monster = GetMonster(buffer_pointer);
+

monster is of type Monster *, and points to somewhere inside your buffer. If you look in your generated header, you'll see it has convenient accessors for all fields, e.g.

+
assert(monster->hp() == 80);
+
assert(monster->mana() == 150); // default
+
assert(strcmp(monster->name()->c_str(), "MyMonster") == 0);
+

These should all be true. Note that we never stored a mana value, so it will return the default.

+

To access sub-objects, in this case the Vec3:

+
auto pos = monster->pos();
+
assert(pos);
+
assert(pos->z() == 3);
+

If we had not set the pos field during serialization, it would be NULL.

+

Similarly, we can access elements of the inventory array:

+
auto inv = monster->inventory();
+
assert(inv);
+
assert(inv->Get(9) == 9);
+

Direct memory access

As you can see from the above examples, all elements in a buffer are accessed through generated accessors. This is because everything is stored in little endian format on all platforms (the accessor performs a swap operation on big endian machines), and also because the layout of things is generally not known to the user.

For structs, layout is deterministic and guaranteed to be the same accross platforms (scalars are aligned to their own size, and structs themselves to their largest member), and you are allowed to access this memory directly by using sizeof() and memcpy on the pointer to a struct, or even an array of structs.

To compute offsets to sub-elements of a struct, make sure they are a structs themselves, as then you can use the pointers to figure out the offset without having to hardcode it. This is handy for use of arrays of structs with calls like glVertexAttribPointer in OpenGL or similar APIs.

@@ -100,8 +110,8 @@ assert(inv->Get(9) == 9);

When you're processing large amounts of data from a source you know (e.g. your own generated data on disk), this is acceptable, but when reading data from the network that can potentially have been modified by an attacker, this is undesirable.

For this reason, you can optionally use a buffer verifier before you access the data. This verifier will check all offsets, all sizes of fields, and null termination of strings to ensure that when a buffer is accessed, all reads will end up inside the buffer.

Each root type will have a verification function generated for it, e.g. for Monster, you can call:

-

bool ok = VerifyMonsterBuffer(Verifier(buf, len));

-

if ok is true, the buffer is safe to read.

+
bool ok = VerifyMonsterBuffer(Verifier(buf, len));
+

if ok is true, the buffer is safe to read.

Besides untrusted data, this function may be useful to call in debug mode, as extra insurance against data being corrupted somewhere along the way.

While verifying a buffer isn't "free", it is typically faster than a full traversal (since any scalar data is not actually touched), and since it may cause the buffer to be brought into cache before reading, the actual overhead may be even lower than expected.

In specialized cases where a denial of service attack is possible, the verifier has two additional constructor arguments that allow you to limit the nesting depth and total amount of tables the verifier may encounter before declaring the buffer malformed.

@@ -117,9 +127,11 @@ assert(inv->Get(9) == 9);

This gives you maximum flexibility. You could even opt to support both, i.e. check for both files, and regenerate the binary from text when required, otherwise just load the binary.

This option is currently only available for C++, or Java through JNI.

As mentioned in the section "Building" above, this technique requires you to link a few more files into your program, and you'll want to include flatbuffers/idl.h.

-

Load text (either a schema or json) into an in-memory buffer (there is a convenient LoadFile() utility function in flatbuffers/util.h if you wish). Construct a parser: 

flatbuffers::Parser parser;
-

Now you can parse any number of text files in sequence: 

parser.Parse(text_file.c_str());
-

This works similarly to how the command-line compiler works: a sequence of files parsed by the same Parser object allow later files to reference definitions in earlier files. Typically this means you first load a schema file (which populates Parser with definitions), followed by one or more JSON files.

+

Load text (either a schema or json) into an in-memory buffer (there is a convenient LoadFile() utility function in flatbuffers/util.h if you wish). Construct a parser:

+
flatbuffers::Parser parser;
+

Now you can parse any number of text files in sequence:

+
parser.Parse(text_file.c_str());
+

This works similarly to how the command-line compiler works: a sequence of files parsed by the same Parser object allow later files to reference definitions in earlier files. Typically this means you first load a schema file (which populates Parser with definitions), followed by one or more JSON files.

As optional argument to Parse, you may specify a null-terminated list of include paths. If not specified, any include statements try to resolve from the current directory.

If there were any parsing errors, Parse will return false, and Parser::err contains a human readable error string with a line number etc, which you should present to the creator of that file.

After each JSON file, the Parser::fbb member variable is the FlatBufferBuilder that contains the binary buffer version of that file, that you can access as described above.

diff --git a/docs/html/md__go_usage.html b/docs/html/md__go_usage.html index 5a873b2..f9c29c8 100644 --- a/docs/html/md__go_usage.html +++ b/docs/html/md__go_usage.html @@ -54,40 +54,47 @@ $(document).ready(function(){initNavTree('md__go_usage.html','');});

There's experimental support for reading FlatBuffers in Go. Generate code for Go with the -g option to flatc.

-

See go_test.go for an example. You import the generated code, read a FlatBuffer binary file into a []byte, which you pass to the GetRootAsMonster function: 

import (
-   example "MyGame/Example"
-   flatbuffers "github.com/google/flatbuffers/go"
-
-   io/ioutil
-)
-
-buf, err := ioutil.ReadFile("monster.dat")
-// handle err
-monster := example.GetRootAsMonster(buf, 0)
-

Now you can access values like this: 

hp := monster.Hp()
-pos := monster.Pos(nil)
-

Note that whenever you access a new object like in the Pos example above, a new temporary accessor object gets created. If your code is very performance sensitive (you iterate through a lot of objects), you can replace nil with a pointer to a Vec3 object you've already created. This allows you to reuse it across many calls and reduce the amount of object allocation (and thus garbage collection) your program does.

-

To access vectors you pass an extra index to the vector field accessor. Then a second method with the same name suffixed by Length let's you know the number of elements you can access: 

for i := 0; i < monster.InventoryLength(); i++ {
-    monster.Inventory(i) // do something here
-}
-

You can also construct these buffers in Go using the functions found in the generated code, and the FlatBufferBuilder class: 

builder := flatbuffers.NewBuilder(0)
-

Create strings: 

str := builder.CreateString("MyMonster")
-

Create a table with a struct contained therein: 

example.MonsterStart(builder)
-example.MonsterAddPos(builder, example.CreateVec3(builder, 1.0, 2.0, 3.0, 3.0, 4, 5, 6))
-example.MonsterAddHp(builder, 80)
-example.MonsterAddName(builder, str)
-example.MonsterAddInventory(builder, inv)
-example.MonsterAddTest_Type(builder, 1)
-example.MonsterAddTest(builder, mon2)
-example.MonsterAddTest4(builder, test4s)
-mon := example.MonsterEnd(builder)
-

Unlike C++, Go does not support table creation functions like 'createMonster()'. This is to create the buffer without using temporary object allocation (since the Vec3 is an inline component of Monster, it has to be created right where it is added, whereas the name and the inventory are not inline). Structs do have convenient methods that allow you to construct them in one call. These also have arguments for nested structs, e.g. if a struct has a field a and a nested struct field b (which has fields c and d), then the arguments will be a, c and d.

-

Vectors also use this start/end pattern to allow vectors of both scalar types and structs: 

example.MonsterStartInventoryVector(builder, 5)
-for i := 4; i >= 0; i-- {
-    builder.PrependByte(byte(i))
-}
-inv := builder.EndVector(5)
-

The generated method 'StartInventoryVector' is provided as a convenience function which calls 'StartVector' with the correct element size of the vector type which in this case is 'ubyte' or 1 byte per vector element. You pass the number of elements you want to write. You write the elements backwards since the buffer is being constructed back to front.

+

See go_test.go for an example. You import the generated code, read a FlatBuffer binary file into a []byte, which you pass to the GetRootAsMonster function:

+
import (
+
example "MyGame/Example"
+
flatbuffers "github.com/google/flatbuffers/go"
+
+
io/ioutil
+
)
+
+
buf, err := ioutil.ReadFile("monster.dat")
+
// handle err
+
monster := example.GetRootAsMonster(buf, 0)
+

Now you can access values like this:

+
hp := monster.Hp()
+
pos := monster.Pos(nil)
+

Note that whenever you access a new object like in the Pos example above, a new temporary accessor object gets created. If your code is very performance sensitive (you iterate through a lot of objects), you can replace nil with a pointer to a Vec3 object you've already created. This allows you to reuse it across many calls and reduce the amount of object allocation (and thus garbage collection) your program does.

+

To access vectors you pass an extra index to the vector field accessor. Then a second method with the same name suffixed by Length let's you know the number of elements you can access:

+
for i := 0; i < monster.InventoryLength(); i++ {
+
monster.Inventory(i) // do something here
+
}
+

You can also construct these buffers in Go using the functions found in the generated code, and the FlatBufferBuilder class:

+
builder := flatbuffers.NewBuilder(0)
+

Create strings:

+
str := builder.CreateString("MyMonster")
+

Create a table with a struct contained therein:

+
example.MonsterStart(builder)
+
example.MonsterAddPos(builder, example.CreateVec3(builder, 1.0, 2.0, 3.0, 3.0, 4, 5, 6))
+
example.MonsterAddHp(builder, 80)
+
example.MonsterAddName(builder, str)
+
example.MonsterAddInventory(builder, inv)
+
example.MonsterAddTest_Type(builder, 1)
+
example.MonsterAddTest(builder, mon2)
+
example.MonsterAddTest4(builder, test4s)
+
mon := example.MonsterEnd(builder)
+

Unlike C++, Go does not support table creation functions like 'createMonster()'. This is to create the buffer without using temporary object allocation (since the Vec3 is an inline component of Monster, it has to be created right where it is added, whereas the name and the inventory are not inline). Structs do have convenient methods that allow you to construct them in one call. These also have arguments for nested structs, e.g. if a struct has a field a and a nested struct field b (which has fields c and d), then the arguments will be a, c and d.

+

Vectors also use this start/end pattern to allow vectors of both scalar types and structs:

+
example.MonsterStartInventoryVector(builder, 5)
+
for i := 4; i >= 0; i-- {
+
builder.PrependByte(byte(i))
+
}
+
inv := builder.EndVector(5)
+

The generated method 'StartInventoryVector' is provided as a convenience function which calls 'StartVector' with the correct element size of the vector type which in this case is 'ubyte' or 1 byte per vector element. You pass the number of elements you want to write. You write the elements backwards since the buffer is being constructed back to front.

There are Prepend functions for all the scalar types. You use PrependUOffset for any previously constructed objects (such as other tables, strings, vectors). For structs, you use the appropriate create function in-line, as shown above in the Monster example.

Text Parsing

There currently is no support for parsing text (Schema's and JSON) directly from Go, though you could use the C++ parser through cgo. Please see the C++ documentation for more on text parsing.

diff --git a/docs/html/md__java_usage.html b/docs/html/md__java_usage.html index 5ba7342..161d037 100644 --- a/docs/html/md__java_usage.html +++ b/docs/html/md__java_usage.html @@ -54,41 +54,51 @@ $(document).ready(function(){initNavTree('md__java_usage.html','');});

FlatBuffers supports reading and writing binary FlatBuffers in Java. Generate code for Java with the -j option to flatc.

-

See javaTest.java for an example. Essentially, you read a FlatBuffer binary file into a byte[], which you then turn into a ByteBuffer, which you pass to the getRootAsMyRootType function: 

ByteBuffer bb = ByteBuffer.wrap(data);
-Monster monster = Monster.getRootAsMonster(bb);
-

Now you can access values much like C++: 

short hp = monster.hp();
-Vec3 pos = monster.pos();
-

Note that whenever you access a new object like in the pos example above, a new temporary accessor object gets created. If your code is very performance sensitive (you iterate through a lot of objects), there's a second pos() method to which you can pass a Vec3 object you've already created. This allows you to reuse it across many calls and reduce the amount of object allocation (and thus garbage collection) your program does.

+

See javaTest.java for an example. Essentially, you read a FlatBuffer binary file into a byte[], which you then turn into a ByteBuffer, which you pass to the getRootAsMyRootType function:

+
ByteBuffer bb = ByteBuffer.wrap(data);
+
Monster monster = Monster.getRootAsMonster(bb);
+

Now you can access values much like C++:

+
short hp = monster.hp();
+
Vec3 pos = monster.pos();
+

Note that whenever you access a new object like in the pos example above, a new temporary accessor object gets created. If your code is very performance sensitive (you iterate through a lot of objects), there's a second pos() method to which you can pass a Vec3 object you've already created. This allows you to reuse it across many calls and reduce the amount of object allocation (and thus garbage collection) your program does.

Java does not support unsigned scalars. This means that any unsigned types you use in your schema will actually be represented as a signed value. This means all bits are still present, but may represent a negative value when used. For example, to read a byte b as an unsigned number, you can do: (short)(b & 0xFF)

The default string accessor (e.g. monster.name()) currently always create a new Java String when accessed, since FlatBuffer's UTF-8 strings can't be used in-place by String. Alternatively, use monster.nameAsByteBuffer() which returns a ByteBuffer referring to the UTF-8 data in the original ByteBuffer, which is much more efficient. The ByteBuffer's position points to the first character, and its limit to just after the last.

-

Vector access is also a bit different from C++: you pass an extra index to the vector field accessor. Then a second method with the same name suffixed by Length let's you know the number of elements you can access: 

for (int i = 0; i < monster.inventoryLength(); i++)
-    monster.inventory(i); // do something here
-

Alternatively, much like strings, you can use monster.inventoryAsByteBuffer() to get a ByteBuffer referring to the whole vector. Use ByteBuffer methods like asFloatBuffer to get specific views if needed.

-

If you specified a file_indentifier in the schema, you can query if the buffer is of the desired type before accessing it using: 

if (Monster.MonsterBufferHasIdentifier(bb)) ...
-

Buffer construction in Java

-

You can also construct these buffers in Java using the static methods found in the generated code, and the FlatBufferBuilder class: 

FlatBufferBuilder fbb = new FlatBufferBuilder();
-

Create strings: 

int str = fbb.createString("MyMonster");
-

Create a table with a struct contained therein: 

Monster.startMonster(fbb);
-Monster.addPos(fbb, Vec3.createVec3(fbb, 1.0f, 2.0f, 3.0f, 3.0, (byte)4, (short)5, (byte)6));
-Monster.addHp(fbb, (short)80);
-Monster.addName(fbb, str);
-Monster.addInventory(fbb, inv);
-Monster.addTest_type(fbb, (byte)1);
-Monster.addTest(fbb, mon2);
-Monster.addTest4(fbb, test4s);
-int mon = Monster.endMonster(fbb);
-

For some simpler types, you can use a convenient create function call that allows you to construct tables in one function call. This example definition however contains an inline struct field, so we have to create the table manually. This is to create the buffer without using temporary object allocation.

+

Vector access is also a bit different from C++: you pass an extra index to the vector field accessor. Then a second method with the same name suffixed by Length let's you know the number of elements you can access:

+
for (int i = 0; i < monster.inventoryLength(); i++)
+
monster.inventory(i); // do something here
+

Alternatively, much like strings, you can use monster.inventoryAsByteBuffer() to get a ByteBuffer referring to the whole vector. Use ByteBuffer methods like asFloatBuffer to get specific views if needed.

+

If you specified a file_indentifier in the schema, you can query if the buffer is of the desired type before accessing it using:

+
if (Monster.MonsterBufferHasIdentifier(bb)) ...
+

Buffer construction in Java

+

You can also construct these buffers in Java using the static methods found in the generated code, and the FlatBufferBuilder class:

+
FlatBufferBuilder fbb = new FlatBufferBuilder();
+

Create strings:

+
int str = fbb.createString("MyMonster");
+

Create a table with a struct contained therein:

+
Monster.startMonster(fbb);
+
Monster.addPos(fbb, Vec3.createVec3(fbb, 1.0f, 2.0f, 3.0f, 3.0, (byte)4, (short)5, (byte)6));
+
Monster.addHp(fbb, (short)80);
+
Monster.addName(fbb, str);
+
Monster.addInventory(fbb, inv);
+
Monster.addTest_type(fbb, (byte)1);
+
Monster.addTest(fbb, mon2);
+
Monster.addTest4(fbb, test4s);
+
int mon = Monster.endMonster(fbb);
+

For some simpler types, you can use a convenient create function call that allows you to construct tables in one function call. This example definition however contains an inline struct field, so we have to create the table manually. This is to create the buffer without using temporary object allocation.

It's important to understand that fields that are structs are inline (like Vec3 above), and MUST thus be created between the start and end calls of a table. Everything else (other tables, strings, vectors) MUST be created before the start of the table they are referenced in.

Structs do have convenient methods that even have arguments for nested structs.

As you can see, references to other objects (e.g. the string above) are simple ints, and thus do not have the type-safety of the Offset type in C++. Extra case must thus be taken that you set the right offset on the right field.

-

Vectors can be created from the corresponding Java array like so: 

int inv = Monster.createInventoryVector(fbb, new byte[] { 0, 1, 2, 3, 4 });
-

This works for arrays of scalars and (int) offsets to strings/tables, but not structs. If you want to write structs, or what you want to write does not sit in an array, you can also use the start/end pattern: 

Monster.startInventoryVector(fbb, 5);
-for (byte i = 4; i >=0; i--) fbb.addByte(i);
-int inv = fbb.endVector();
-

You can use the generated method startInventoryVector to conveniently call startVector with the right element size. You pass the number of elements you want to write. Note how you write the elements backwards since the buffer is being constructed back to front.

+

Vectors can be created from the corresponding Java array like so:

+
int inv = Monster.createInventoryVector(fbb, new byte[] { 0, 1, 2, 3, 4 });
+

This works for arrays of scalars and (int) offsets to strings/tables, but not structs. If you want to write structs, or what you want to write does not sit in an array, you can also use the start/end pattern:

+
Monster.startInventoryVector(fbb, 5);
+
for (byte i = 4; i >=0; i--) fbb.addByte(i);
+
int inv = fbb.endVector();
+

You can use the generated method startInventoryVector to conveniently call startVector with the right element size. You pass the number of elements you want to write. Note how you write the elements backwards since the buffer is being constructed back to front.

There are add functions for all the scalar types. You use addOffset for any previously constructed objects (such as other tables, strings, vectors). For structs, you use the appropriate create function in-line, as shown above in the Monster example.

-

To finish the buffer, call: 

Monster.finishMonsterBuffer(fbb, mon);
-

The buffer is now ready to be transmitted. It is contained in the ByteBuffer which you can obtain from fbb.dataBuffer(). Importantly, the valid data does not start from offset 0 in this buffer, but from fbb.dataBuffer().position() (this is because the data was built backwards in memory). It ends at fbb.capacity().

+

To finish the buffer, call:

+
Monster.finishMonsterBuffer(fbb, mon);
+

The buffer is now ready to be transmitted. It is contained in the ByteBuffer which you can obtain from fbb.dataBuffer(). Importantly, the valid data does not start from offset 0 in this buffer, but from fbb.dataBuffer().position() (this is because the data was built backwards in memory). It ends at fbb.capacity().

Text Parsing

There currently is no support for parsing text (Schema's and JSON) directly from Java, though you could use the C++ parser through JNI. Please see the C++ documentation for more on text parsing.

diff --git a/docs/source/CppUsage.md b/docs/source/CppUsage.md index 81aaa8d..9721c6c 100755 --- a/docs/source/CppUsage.md +++ b/docs/source/CppUsage.md @@ -12,17 +12,21 @@ your program by including the header. As noted, this header relies on To start creating a buffer, create an instance of `FlatBufferBuilder` which will contain the buffer as it grows: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp} FlatBufferBuilder fbb; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Before we serialize a Monster, we need to first serialize any objects that are contained there-in, i.e. we serialize the data tree using depth first, pre-order traversal. This is generally easy to do on any tree structures. For example: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp} auto name = fbb.CreateString("MyMonster"); unsigned char inv[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }; auto inventory = fbb.CreateVector(inv, 10); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `CreateString` and `CreateVector` serialize these two built-in datatypes, and return offsets into the serialized data indicating where @@ -38,7 +42,9 @@ correct type below. To create a vector of struct objects (which will be stored as contiguous memory in the buffer, use `CreateVectorOfStructs` instead. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp} Vec3 vec(1, 2, 3); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `Vec3` is the first example of code from our generated header. Structs (unlike tables) translate to simple structs in C++, so @@ -47,7 +53,9 @@ we can construct them in a familiar way. We have now serialized the non-scalar components of of the monster example, so we could create the monster something like this: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp} auto mloc = CreateMonster(fbb, &vec, 150, 80, name, inventory, Color_Red, 0, Any_NONE); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Note that we're passing `150` for the `mana` field, which happens to be the default value: this means the field will not actually be written to the buffer, @@ -67,12 +75,14 @@ If you want even more control over this (i.e. skip fields even when they are not default), instead of the convenient `CreateMonster` call we can also build the object field-by-field manually: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp} MonsterBuilder mb(fbb); mb.add_pos(&vec); mb.add_hp(80); mb.add_name(name); mb.add_inventory(inventory); auto mloc = mb.Finish(); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We start with a temporary helper class `MonsterBuilder` (which is defined in our generated code also), then call the various `add_` @@ -88,7 +98,9 @@ Regardless of whether you used `CreateMonster` or `MonsterBuilder`, you now have an offset to the root of your data, and you can finish the buffer using: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp} FinishMonsterBuffer(fbb, mloc); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The buffer is now ready to be stored somewhere, sent over the network, be compressed, or whatever you'd like to do with it. You can access the @@ -103,33 +115,41 @@ the code above, that also includes the reading code below. If you've received a buffer from somewhere (disk, network, etc.) you can directly start traversing it using: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp} auto monster = GetMonster(buffer_pointer); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `monster` is of type `Monster *`, and points to somewhere inside your buffer. If you look in your generated header, you'll see it has convenient accessors for all fields, e.g. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp} assert(monster->hp() == 80); assert(monster->mana() == 150); // default assert(strcmp(monster->name()->c_str(), "MyMonster") == 0); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ These should all be true. Note that we never stored a `mana` value, so it will return the default. To access sub-objects, in this case the `Vec3`: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp} auto pos = monster->pos(); assert(pos); assert(pos->z() == 3); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If we had not set the `pos` field during serialization, it would be `NULL`. Similarly, we can access elements of the inventory array: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp} auto inv = monster->inventory(); assert(inv); assert(inv->Get(9) == 9); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ### Direct memory access @@ -175,7 +195,9 @@ is accessed, all reads will end up inside the buffer. Each root type will have a verification function generated for it, e.g. for `Monster`, you can call: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp} bool ok = VerifyMonsterBuffer(Verifier(buf, len)); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ if `ok` is true, the buffer is safe to read. @@ -237,11 +259,15 @@ Load text (either a schema or json) into an in-memory buffer (there is a convenient `LoadFile()` utility function in `flatbuffers/util.h` if you wish). Construct a parser: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp} flatbuffers::Parser parser; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Now you can parse any number of text files in sequence: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp} parser.Parse(text_file.c_str()); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This works similarly to how the command-line compiler works: a sequence of files parsed by the same `Parser` object allow later files to diff --git a/docs/source/GoUsage.md b/docs/source/GoUsage.md index 0c2370b..d70fc28 100644 --- a/docs/source/GoUsage.md +++ b/docs/source/GoUsage.md @@ -7,6 +7,7 @@ See `go_test.go` for an example. You import the generated code, read a FlatBuffer binary file into a `[]byte`, which you pass to the `GetRootAsMonster` function: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.go} import ( example "MyGame/Example" flatbuffers "github.com/google/flatbuffers/go" @@ -17,11 +18,14 @@ FlatBuffer binary file into a `[]byte`, which you pass to the buf, err := ioutil.ReadFile("monster.dat") // handle err monster := example.GetRootAsMonster(buf, 0) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Now you can access values like this: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.go} hp := monster.Hp() pos := monster.Pos(nil) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Note that whenever you access a new object like in the `Pos` example above, a new temporary accessor object gets created. If your code is very performance @@ -34,21 +38,28 @@ To access vectors you pass an extra index to the vector field accessor. Then a second method with the same name suffixed by `Length` let's you know the number of elements you can access: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.go} for i := 0; i < monster.InventoryLength(); i++ { monster.Inventory(i) // do something here } +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can also construct these buffers in Go using the functions found in the generated code, and the FlatBufferBuilder class: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.go} builder := flatbuffers.NewBuilder(0) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Create strings: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.go} str := builder.CreateString("MyMonster") +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Create a table with a struct contained therein: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.go} example.MonsterStart(builder) example.MonsterAddPos(builder, example.CreateVec3(builder, 1.0, 2.0, 3.0, 3.0, 4, 5, 6)) example.MonsterAddHp(builder, 80) @@ -58,6 +69,7 @@ Create a table with a struct contained therein: example.MonsterAddTest(builder, mon2) example.MonsterAddTest4(builder, test4s) mon := example.MonsterEnd(builder) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Unlike C++, Go does not support table creation functions like 'createMonster()'. This is to create the buffer without @@ -72,11 +84,13 @@ will be `a`, `c` and `d`. Vectors also use this start/end pattern to allow vectors of both scalar types and structs: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.go} example.MonsterStartInventoryVector(builder, 5) for i := 4; i >= 0; i-- { builder.PrependByte(byte(i)) } inv := builder.EndVector(5) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The generated method 'StartInventoryVector' is provided as a convenience function which calls 'StartVector' with the correct element size of the vector diff --git a/docs/source/JavaUsage.md b/docs/source/JavaUsage.md index 7b0af0c..210cd87 100755 --- a/docs/source/JavaUsage.md +++ b/docs/source/JavaUsage.md @@ -7,13 +7,17 @@ See `javaTest.java` for an example. Essentially, you read a FlatBuffer binary file into a `byte[]`, which you then turn into a `ByteBuffer`, which you pass to the `getRootAsMyRootType` function: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} ByteBuffer bb = ByteBuffer.wrap(data); Monster monster = Monster.getRootAsMonster(bb); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Now you can access values much like C++: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} short hp = monster.hp(); Vec3 pos = monster.pos(); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Note that whenever you access a new object like in the `pos` example above, a new temporary accessor object gets created. If your code is very performance @@ -39,8 +43,10 @@ Vector access is also a bit different from C++: you pass an extra index to the vector field accessor. Then a second method with the same name suffixed by `Length` let's you know the number of elements you can access: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} for (int i = 0; i < monster.inventoryLength(); i++) monster.inventory(i); // do something here +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Alternatively, much like strings, you can use `monster.inventoryAsByteBuffer()` to get a `ByteBuffer` referring to the whole vector. Use `ByteBuffer` methods @@ -49,7 +55,9 @@ like `asFloatBuffer` to get specific views if needed. If you specified a file_indentifier in the schema, you can query if the buffer is of the desired type before accessing it using: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} if (Monster.MonsterBufferHasIdentifier(bb)) ... +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ## Buffer construction in Java @@ -57,14 +65,19 @@ buffer is of the desired type before accessing it using: You can also construct these buffers in Java using the static methods found in the generated code, and the FlatBufferBuilder class: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} FlatBufferBuilder fbb = new FlatBufferBuilder(); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Create strings: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} int str = fbb.createString("MyMonster"); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Create a table with a struct contained therein: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} Monster.startMonster(fbb); Monster.addPos(fbb, Vec3.createVec3(fbb, 1.0f, 2.0f, 3.0f, 3.0, (byte)4, (short)5, (byte)6)); Monster.addHp(fbb, (short)80); @@ -74,6 +87,7 @@ Create a table with a struct contained therein: Monster.addTest(fbb, mon2); Monster.addTest4(fbb, test4s); int mon = Monster.endMonster(fbb); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For some simpler types, you can use a convenient `create` function call that allows you to construct tables in one function call. This example definition @@ -94,15 +108,19 @@ case must thus be taken that you set the right offset on the right field. Vectors can be created from the corresponding Java array like so: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} int inv = Monster.createInventoryVector(fbb, new byte[] { 0, 1, 2, 3, 4 }); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This works for arrays of scalars and (int) offsets to strings/tables, but not structs. If you want to write structs, or what you want to write does not sit in an array, you can also use the start/end pattern: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} Monster.startInventoryVector(fbb, 5); for (byte i = 4; i >=0; i--) fbb.addByte(i); int inv = fbb.endVector(); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can use the generated method `startInventoryVector` to conveniently call `startVector` with the right element size. You pass the number of @@ -116,7 +134,9 @@ above in the `Monster` example. To finish the buffer, call: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.java} Monster.finishMonsterBuffer(fbb, mon); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The buffer is now ready to be transmitted. It is contained in the `ByteBuffer` which you can obtain from `fbb.dataBuffer()`. Importantly, the valid data does -- 2.7.4