3 This section is entirely optional for the use of FlatBuffers. In normal
4 usage, you should never need the information contained herein. If you're
5 interested however, it should give you more of an appreciation of why
6 FlatBuffers is both efficient and convenient.
10 A FlatBuffer is a binary file and in-memory format consisting mostly of
11 scalars of various sizes, all aligned to their own size. Each scalar is
12 also always represented in little-endian format, as this corresponds to
13 all commonly used CPUs today. FlatBuffers will also work on big-endian
14 machines, but will be slightly slower because of additional
17 On purpose, the format leaves a lot of details about where exactly
18 things live in memory undefined, e.g. fields in a table can have any
19 order, and objects to some extend can be stored in many orders. This is
20 because the format doesn't need this information to be efficient, and it
21 leaves room for optimization and extension (for example, fields can be
22 packed in a way that is most compact). Instead, the format is defined in
23 terms of offsets and adjacency only. This may mean two different
24 implementations may produce different binaries given the same input
25 values, and this is perfectly valid.
27 ### Format identification
29 The format also doesn't contain information for format identification
30 and versioning, which is also by design. FlatBuffers is a statically typed
31 system, meaning the user of a buffer needs to know what kind of buffer
32 it is. FlatBuffers can of course be wrapped inside other containers
33 where needed, or you can use its union feature to dynamically identify
34 multiple possible sub-objects stored. Additionally, it can be used
35 together with the schema parser if full reflective capabilities are
38 Versioning is something that is intrinsically part of the format (the
39 optionality / extensibility of fields), so the format itself does not
40 need a version number (it's a meta-format, in a sense). We're hoping
41 that this format can accommodate all data needed. If format breaking
42 changes are ever necessary, it would become a new kind of format rather
43 than just a variation.
47 The most important and generic offset type (see `flatbuffers.h`) is
48 `uoffset_t`, which is currently always a `uint32_t`, and is used to
49 refer to all tables/unions/strings/vectors (these are never stored
51 intentional, since we want to keep the format binary compatible between
52 32 and 64bit systems, and a 64bit offset would bloat the size for almost
53 all uses. A version of this format with 64bit (or 16bit) offsets is easy to set
54 when needed. Unsigned means they can only point in one direction, which
55 typically is forward (towards a higher memory location). Any backwards
56 offsets will be explicitly marked as such.
58 The format starts with an `uoffset_t` to the root object in the buffer.
60 We have two kinds of objects, structs and tables.
64 These are the simplest, and as mentioned, intended for simple data that
65 benefits from being extra efficient and doesn't need versioning /
66 extensibility. They are always stored inline in their parent (a struct,
67 table, or vector) for maximum compactness. Structs define a consistent
68 memory layout where all components are aligned to their size, and
69 structs aligned to their largest scalar member. This is done independent
70 of the alignment rules of the underlying compiler to guarantee a cross
71 platform compatible layout. This layout is then enforced in the generated
76 Unlike structs, these are not stored in inline in their parent, but are
77 referred to by offset.
79 They start with an `soffset_t` to a vtable. This is a signed version of
80 `uoffset_t`, since vtables may be stored anywhere relative to the object.
81 This offset is substracted (not added) from the object start to arrive at
82 the vtable start. This offset is followed by all the
83 fields as aligned scalars (or offsets). Unlike structs, not all fields
84 need to be present. There is no set order and layout.
86 To be able to access fields regardless of these uncertainties, we go
87 through a vtable of offsets. Vtables are shared between any objects that
88 happen to have the same vtable values.
90 The elements of a vtable are all of type `voffset_t`, which is
91 a `uint16_t`. The first element is the size of the vtable in bytes,
92 including the size element. The second one is the size of the object, in bytes
93 (including the vtable offset). This size could be used for streaming, to know
94 how many bytes to read to be able to access all *inline* fields of the object.
95 The remaining elements are the N offsets, where N is the amount of fields
96 declared in the schema when the code that constructed this buffer was
97 compiled (thus, the size of the table is N + 2).
99 All accessor functions in the generated code for tables contain the
100 offset into this table as a constant. This offset is checked against the
101 first field (the number of elements), to protect against newer code
102 reading older data. If this offset is out of range, or the vtable entry
103 is 0, that means the field is not present in this object, and the
104 default value is return. Otherwise, the entry is used as offset to the
107 ### Strings and Vectors
109 Strings are simply a vector of bytes, and are always
110 null-terminated. Vectors are stored as contiguous aligned scalar
111 elements prefixed by a 32bit element count (not including any
112 null termination). Neither is stored inline in their parent, but are referred to
117 The current implementation constructs these buffers backwards (starting
118 at the highest memory address of the buffer), since
119 that significantly reduces the amount of bookkeeping and simplifies the
124 Here's an example of the code that gets generated for the `samples/monster.fbs`.
125 What follows is the entire file, broken up by comments:
127 // automatically generated, do not modify
129 #include "flatbuffers/flatbuffers.h"
134 Nested namespace support.
142 inline const char **EnumNamesColor() {
143 static const char *names[] = { "Red", "Green", "Blue", nullptr };
147 inline const char *EnumNameColor(int e) { return EnumNamesColor()[e]; }
149 Enums and convenient reverse lookup.
156 inline const char **EnumNamesAny() {
157 static const char *names[] = { "NONE", "Monster", nullptr };
161 inline const char *EnumNameAny(int e) { return EnumNamesAny()[e]; }
163 Unions share a lot with enums.
168 Predeclare all data types since circular references between types are allowed
169 (circular references between object are not, though).
171 MANUALLY_ALIGNED_STRUCT(4) Vec3 {
178 Vec3(float x, float y, float z)
179 : x_(flatbuffers::EndianScalar(x)), y_(flatbuffers::EndianScalar(y)), z_(flatbuffers::EndianScalar(z)) {}
181 float x() const { return flatbuffers::EndianScalar(x_); }
182 float y() const { return flatbuffers::EndianScalar(y_); }
183 float z() const { return flatbuffers::EndianScalar(z_); }
185 STRUCT_END(Vec3, 12);
187 These ugly macros do a couple of things: they turn off any padding the compiler
188 might normally do, since we add padding manually (though none in this example),
189 and they enforce alignment chosen by FlatBuffers. This ensures the layout of
190 this struct will look the same regardless of compiler and platform. Note that
191 the fields are private: this is because these store little endian scalars
192 regardless of platform (since this is part of the serialized data).
193 `EndianScalar` then converts back and forth, which is a no-op on all current
194 mobile and desktop platforms, and a single machine instruction on the few
195 remaining big endian platforms.
197 struct Monster : private flatbuffers::Table {
198 const Vec3 *pos() const { return GetStruct<const Vec3 *>(4); }
199 int16_t mana() const { return GetField<int16_t>(6, 150); }
200 int16_t hp() const { return GetField<int16_t>(8, 100); }
201 const flatbuffers::String *name() const { return GetPointer<const flatbuffers::String *>(10); }
202 const flatbuffers::Vector<uint8_t> *inventory() const { return GetPointer<const flatbuffers::Vector<uint8_t> *>(14); }
203 int8_t color() const { return GetField<int8_t>(16, 2); }
206 Tables are a bit more complicated. A table accessor struct is used to point at
207 the serialized data for a table, which always starts with an offset to its
208 vtable. It derives from `Table`, which contains the `GetField` helper functions.
209 GetField takes a vtable offset, and a default value. It will look in the vtable
210 at that offset. If the offset is out of bounds (data from an older version) or
211 the vtable entry is 0, the field is not present and the default is returned.
212 Otherwise, it uses the entry as an offset into the table to locate the field.
214 struct MonsterBuilder {
215 flatbuffers::FlatBufferBuilder &fbb_;
216 flatbuffers::uoffset_t start_;
217 void add_pos(const Vec3 *pos) { fbb_.AddStruct(4, pos); }
218 void add_mana(int16_t mana) { fbb_.AddElement<int16_t>(6, mana, 150); }
219 void add_hp(int16_t hp) { fbb_.AddElement<int16_t>(8, hp, 100); }
220 void add_name(flatbuffers::Offset<flatbuffers::String> name) { fbb_.AddOffset(10, name); }
221 void add_inventory(flatbuffers::Offset<flatbuffers::Vector<uint8_t>> inventory) { fbb_.AddOffset(14, inventory); }
222 void add_color(int8_t color) { fbb_.AddElement<int8_t>(16, color, 2); }
223 MonsterBuilder(flatbuffers::FlatBufferBuilder &_fbb) : fbb_(_fbb) { start_ = fbb_.StartTable(); }
224 flatbuffers::Offset<Monster> Finish() { return flatbuffers::Offset<Monster>(fbb_.EndTable(start_, 7)); }
227 `MonsterBuilder` is the base helper struct to construct a table using a
228 `FlatBufferBuilder`. You can add the fields in any order, and the `Finish`
229 call will ensure the correct vtable gets generated.
231 inline flatbuffers::Offset<Monster> CreateMonster(flatbuffers::FlatBufferBuilder &_fbb, const Vec3 *pos, int16_t mana, int16_t hp, flatbuffers::Offset<flatbuffers::String> name, flatbuffers::Offset<flatbuffers::Vector<uint8_t>> inventory, int8_t color) {
232 MonsterBuilder builder_(_fbb);
233 builder_.add_inventory(inventory);
234 builder_.add_name(name);
235 builder_.add_pos(pos);
237 builder_.add_mana(mana);
238 builder_.add_color(color);
239 return builder_.Finish();
242 `CreateMonster` is a convenience function that calls all functions in
243 `MonsterBuilder` above for you. Note that if you pass values which are
244 defaults as arguments, it will not actually construct that field, so
245 you can probably use this function instead of the builder class in
248 inline const Monster *GetMonster(const void *buf) { return flatbuffers::GetRoot<Monster>(buf); }
250 This function is only generated for the root table type, to be able to
251 start traversing a FlatBuffer from a raw buffer pointer.
253 }; // namespace MyGame
254 }; // namespace Sample
256 ### Encoding example.
258 Below is a sample encoding for the following JSON corresponding to the above
261 { pos: { x: 1, y: 2, z: 3 }, name: "fred", hp: 50 }
263 Resulting in this binary buffer:
265 // Start of the buffer:
266 uint32_t 20 // Offset to the root table.
268 // Start of the vtable. Not shared in this example, but could be:
269 uint16_t 16 // Size of table, starting from here.
270 uint16_t 22 // Size of object inline data.
271 uint16_t 4, 0, 20, 16, 0, 0 // Offsets to fields from start of (root) table, 0 for not present.
273 // Start of the root table:
274 int32_t 16 // Offset to vtable used (default negative direction)
275 float 1, 2, 3 // the Vec3 struct, inline.
276 uint32_t 8 // Offset to the name string.
277 int16_t 50 // hp field.
278 int16_t 0 // Padding for alignment.
280 // Start of name string:
281 uint32_t 4 // Length of string.
282 int8_t 'f', 'r', 'e', 'd', 0, 0, 0, 0 // Text + 0 termination + padding.
284 Note that this not the only possible encoding, since the writer has some
285 flexibility in which of the children of root object to write first (though in
286 this case there's only one string), and what order to write the fields in.
287 Different orders may also cause different alignments to happen.