1 # SPDX-License-Identifier: GPL-2.0+
2 # Copyright (c) 2018 Google, Inc
3 # Written by Simon Glass <sjg@chromium.org>
5 """Entry-type module for sections (groups of entries)
7 Sections are entries which can contain other entries. This allows hierarchical
11 from collections import OrderedDict
12 import concurrent.futures
16 from binman import comp_util
17 from binman.entry import Entry
18 from binman import state
19 from dtoc import fdt_util
20 from patman import tools
21 from patman import tout
22 from patman.tools import to_hex_size
25 class Entry_section(Entry):
26 """Entry that contains other entries
28 A section is an entry which can contain other entries, thus allowing
29 hierarchical images to be created. See 'Sections and hierarchical images'
30 in the binman README for more information.
32 The base implementation simply joins the various entries together, using
33 various rules about alignment, etc.
38 This class can be subclassed to support other file formats which hold
39 multiple entries, such as CBFS. To do this, override the following
40 functions. The documentation here describes what your function should do.
41 For example code, see etypes which subclass `Entry_section`, or `cbfs.py`
42 for a more involved example::
44 $ grep -l \(Entry_section tools/binman/etype/*.py
47 Call `super().ReadNode()`, then read any special properties for the
48 section. Then call `self.ReadEntries()` to read the entries.
50 Binman calls this at the start when reading the image description.
53 Read in the subnodes of the section. This may involve creating entries
54 of a particular etype automatically, as well as reading any special
55 properties in the entries. For each entry, entry.ReadNode() should be
56 called, to read the basic entry properties. The properties should be
57 added to `self._entries[]`, in the correct order, with a suitable name.
59 Binman calls this at the start when reading the image description.
61 BuildSectionData(required)
62 Create the custom file format that you want and return it as bytes.
63 This likely sets up a file header, then loops through the entries,
64 adding them to the file. For each entry, call `entry.GetData()` to
65 obtain the data. If that returns None, and `required` is False, then
66 this method must give up and return None. But if `required` is True then
67 it should assume that all data is valid.
69 Binman calls this when packing the image, to find out the size of
70 everything. It is called again at the end when building the final image.
72 SetImagePos(image_pos):
73 Call `super().SetImagePos(image_pos)`, then set the `image_pos` values
74 for each of the entries. This should use the custom file format to find
75 the `start offset` (and `image_pos`) of each entry. If the file format
76 uses compression in such a way that there is no offset available (other
77 than reading the whole file and decompressing it), then the offsets for
78 affected entries can remain unset (`None`). The size should also be set
81 Binman calls this after the image has been packed, to update the
82 location that all the entries ended up at.
84 ReadChildData(child, decomp, alt_format):
85 The default version of this may be good enough, if you are able to
86 implement SetImagePos() correctly. But that is a bit of a bypass, so
87 you can override this method to read from your custom file format. It
88 should read the entire entry containing the custom file using
89 `super().ReadData(True)`, then parse the file to get the data for the
90 given child, then return that data.
92 If your file format supports compression, the `decomp` argument tells
93 you whether to return the compressed data (`decomp` is False) or to
94 uncompress it first, then return the uncompressed data (`decomp` is
95 True). This is used by the `binman extract -U` option.
97 If your entry supports alternative formats, the alt_format provides the
98 alternative format that the user has selected. Your function should
99 return data in that format. This is used by the 'binman extract -l'
102 Binman calls this when reading in an image, in order to populate all the
103 entries with the data from that image (`binman ls`).
105 WriteChildData(child):
106 Binman calls this after `child.data` is updated, to inform the custom
107 file format about this, in case it needs to do updates.
109 The default version of this does nothing and probably needs to be
110 overridden for the 'binman replace' command to work. Your version should
111 use `child.data` to update the data for that child in the custom file
114 Binman calls this when updating an image that has been read in and in
115 particular to update the data for a particular entry (`binman replace`)
117 Properties / Entry arguments
118 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
120 See :ref:`develop/package/binman:Image description format` for more
124 Default alignment for this section, if no alignment is given in the
128 Pad byte to use when padding
131 True if entries should be sorted by offset, False if they must be
132 in-order in the device tree description
135 Used to build an x86 ROM which ends at 4GB (2^32)
138 Adds a prefix to the name of every entry in the section when writing out
142 Number of bytes before the first entry starts. These effectively adjust
143 the starting offset of entries. For example, if this is 16, then the
144 first entry would start at 16. An entry with offset = 20 would in fact
145 be written at offset 4 in the image file, since the first 16 bytes are
146 skipped when writing.
148 Since a section is also an entry, it inherits all the properies of entries
151 Note that the `allow_missing` member controls whether this section permits
152 external blobs to be missing their contents. The option will produce an
153 image but of course it will not work. It is useful to make sure that
154 Continuous Integration systems can build without the binaries being
155 available. This is set by the `SetAllowMissing()` method, if
156 `--allow-missing` is passed to binman.
158 def __init__(self, section, etype, node, test=False):
160 super().__init__(section, etype, node)
161 self._entries = OrderedDict()
164 self._skip_at_start = None
165 self._end_4gb = False
166 self._ignore_missing = False
169 """Read properties from the section node"""
171 self._pad_byte = fdt_util.GetInt(self._node, 'pad-byte', 0)
172 self._sort = fdt_util.GetBool(self._node, 'sort-by-offset')
173 self._end_4gb = fdt_util.GetBool(self._node, 'end-at-4gb')
174 self._skip_at_start = fdt_util.GetInt(self._node, 'skip-at-start')
177 self.Raise("Section size must be provided when using end-at-4gb")
178 if self._skip_at_start is not None:
179 self.Raise("Provide either 'end-at-4gb' or 'skip-at-start'")
181 self._skip_at_start = 0x100000000 - self.size
183 if self._skip_at_start is None:
184 self._skip_at_start = 0
185 self._name_prefix = fdt_util.GetString(self._node, 'name-prefix')
186 self.align_default = fdt_util.GetInt(self._node, 'align-default', 0)
190 def ReadEntries(self):
191 for node in self._node.subnodes:
192 if node.name.startswith('hash') or node.name.startswith('signature'):
194 entry = Entry.Create(self, node,
195 expanded=self.GetImage().use_expanded,
196 missing_etype=self.GetImage().missing_etype)
198 entry.SetPrefix(self._name_prefix)
199 self._entries[node.name] = entry
201 def _Raise(self, msg):
202 """Raises an error for this section
205 msg (str): Error message to use in the raise string
209 raise ValueError("Section '%s': %s" % (self._node.path, msg))
213 for entry in self._entries.values():
214 fdts.update(entry.GetFdts())
217 def ProcessFdt(self, fdt):
218 """Allow entries to adjust the device tree
220 Some entries need to adjust the device tree for their purposes. This
221 may involve adding or deleting properties.
223 todo = self._entries.values()
224 for passnum in range(3):
227 if not entry.ProcessFdt(fdt):
228 next_todo.append(entry)
233 self.Raise('Internal error: Could not complete processing of Fdt: remaining %s' %
237 def gen_entries(self):
238 super().gen_entries()
239 for entry in self._entries.values():
242 def AddMissingProperties(self, have_image_pos):
243 """Add new properties to the device tree as needed for this entry"""
244 super().AddMissingProperties(have_image_pos)
245 if self.compress != 'none':
246 have_image_pos = False
247 for entry in self._entries.values():
248 entry.AddMissingProperties(have_image_pos)
250 def ObtainContents(self, fake_size=0, skip_entry=None):
251 return self.GetEntryContents(skip_entry=skip_entry)
253 def GetPaddedDataForEntry(self, entry, entry_data):
254 """Get the data for an entry including any padding
256 Gets the entry data and uses the section pad-byte value to add padding
257 before and after as defined by the pad-before and pad-after properties.
258 This does not consider alignment.
261 entry: Entry to check
264 Contents of the entry along with any pad bytes before and
267 pad_byte = (entry._pad_byte if isinstance(entry, Entry_section)
271 # Handle padding before the entry
273 data += tools.get_bytes(self._pad_byte, entry.pad_before)
275 # Add in the actual entry data
278 # Handle padding after the entry
280 data += tools.get_bytes(self._pad_byte, entry.pad_after)
283 data += tools.get_bytes(pad_byte, entry.size - len(data))
285 self.Detail('GetPaddedDataForEntry: size %s' % to_hex_size(self.data))
289 def BuildSectionData(self, required):
290 """Build the contents of a section
292 This places all entries at the right place, dealing with padding before
293 and after entries. It does not do padding for the section itself (the
294 pad-before and pad-after properties in the section items) since that is
295 handled by the parent section.
297 This should be overridden by subclasses which want to build their own
298 data structure for the section.
301 required: True if the data must be present, False if it is OK to
305 Contents of the section (bytes)
307 section_data = bytearray()
309 for entry in self._entries.values():
310 entry_data = entry.GetData(required)
312 # This can happen when this section is referenced from a collection
313 # earlier in the image description. See testCollectionSection().
314 if not required and entry_data is None:
316 data = self.GetPaddedDataForEntry(entry, entry_data)
317 # Handle empty space before the entry
318 pad = (entry.offset or 0) - self._skip_at_start - len(section_data)
320 section_data += tools.get_bytes(self._pad_byte, pad)
322 # Add in the actual entry data
325 self.Detail('GetData: %d entries, total size %#x' %
326 (len(self._entries), len(section_data)))
327 return self.CompressData(section_data)
329 def GetPaddedData(self, data=None):
330 """Get the data for a section including any padding
332 Gets the section data and uses the parent section's pad-byte value to
333 add padding before and after as defined by the pad-before and pad-after
334 properties. If this is a top-level section (i.e. an image), this is the
335 same as GetData(), since padding is not supported.
337 This does not consider alignment.
340 Contents of the section along with any pad bytes before and
343 section = self.section or self
345 data = self.GetData()
346 return section.GetPaddedDataForEntry(self, data)
348 def GetData(self, required=True):
349 """Get the contents of an entry
351 This builds the contents of the section, stores this as the contents of
352 the section and returns it
355 required: True if the data must be present, False if it is OK to
359 bytes content of the section, made up for all all of its subentries.
360 This excludes any padding. If the section is compressed, the
361 compressed data is returned
363 data = self.BuildSectionData(required)
366 self.SetContents(data)
369 def GetOffsets(self):
370 """Handle entries that want to set the offset/size of other entries
372 This calls each entry's GetOffsets() method. If it returns a list
373 of entries to update, it updates them.
375 self.GetEntryOffsets()
378 def ResetForPack(self):
379 """Reset offset/size fields so that packing can be done again"""
380 super().ResetForPack()
381 for entry in self._entries.values():
384 def Pack(self, offset):
385 """Pack all entries into the section"""
389 self._extend_entries()
391 data = self.BuildSectionData(True)
392 self.SetContents(data)
396 offset = super().Pack(offset)
400 def _PackEntries(self):
401 """Pack all entries into the section"""
402 offset = self._skip_at_start
403 for entry in self._entries.values():
404 offset = entry.Pack(offset)
407 def _extend_entries(self):
408 """Extend any entries that are permitted to"""
410 for entry in self._entries.values():
412 exp_entry.extend_to_limit(entry.offset)
414 if entry.extend_size:
417 exp_entry.extend_to_limit(self.size)
419 def _SortEntries(self):
420 """Sort entries by offset"""
421 entries = sorted(self._entries.values(), key=lambda entry: entry.offset)
422 self._entries.clear()
423 for entry in entries:
424 self._entries[entry._node.name] = entry
426 def CheckEntries(self):
427 """Check that entries do not overlap or extend outside the section"""
428 max_size = self.size if self.uncomp_size is None else self.uncomp_size
432 for entry in self._entries.values():
434 if (entry.offset < self._skip_at_start or
435 entry.offset + entry.size > self._skip_at_start +
437 entry.Raise('Offset %#x (%d) size %#x (%d) is outside the '
438 "section '%s' starting at %#x (%d) "
440 (entry.offset, entry.offset, entry.size, entry.size,
441 self._node.path, self._skip_at_start,
442 self._skip_at_start, max_size, max_size))
443 if entry.offset < offset and entry.size:
444 entry.Raise("Offset %#x (%d) overlaps with previous entry '%s' "
445 "ending at %#x (%d)" %
446 (entry.offset, entry.offset, prev_name, offset, offset))
447 offset = entry.offset + entry.size
448 prev_name = entry.GetPath()
450 def WriteSymbols(self, section):
451 """Write symbol values into binary files for access at run time"""
452 for entry in self._entries.values():
453 entry.WriteSymbols(self)
455 def SetCalculatedProperties(self):
456 super().SetCalculatedProperties()
457 for entry in self._entries.values():
458 entry.SetCalculatedProperties()
460 def SetImagePos(self, image_pos):
461 super().SetImagePos(image_pos)
462 if self.compress == 'none':
463 for entry in self._entries.values():
464 entry.SetImagePos(image_pos + self.offset)
466 def ProcessContents(self):
467 sizes_ok_base = super(Entry_section, self).ProcessContents()
469 for entry in self._entries.values():
470 if not entry.ProcessContents():
472 return sizes_ok and sizes_ok_base
474 def WriteMap(self, fd, indent):
475 """Write a map of the section to a .map file
478 fd: File to write the map to
480 Entry.WriteMapLine(fd, indent, self.name, self.offset or 0,
481 self.size, self.image_pos)
482 for entry in self._entries.values():
483 entry.WriteMap(fd, indent + 1)
485 def GetEntries(self):
488 def GetContentsByPhandle(self, phandle, source_entry, required):
489 """Get the data contents of an entry specified by a phandle
491 This uses a phandle to look up a node and and find the entry
492 associated with it. Then it returns the contents of that entry.
494 The node must be a direct subnode of this section.
497 phandle: Phandle to look up (integer)
498 source_entry: Entry containing that phandle (used for error
500 required: True if the data must be present, False if it is OK to
504 data from associated entry (as a string), or None if not found
506 node = self._node.GetFdt().LookupPhandle(phandle)
508 source_entry.Raise("Cannot find node for phandle %d" % phandle)
509 entry = self.FindEntryByNode(node)
511 source_entry.Raise("Cannot find entry for node '%s'" % node.name)
512 return entry.GetData(required)
514 def LookupSymbol(self, sym_name, optional, msg, base_addr, entries=None):
515 """Look up a symbol in an ELF file
517 Looks up a symbol in an ELF file. Only entry types which come from an
518 ELF image can be used by this function.
520 At present the only entry properties supported are:
522 image_pos - 'base_addr' is added if this is not an end-at-4gb image
526 sym_name: Symbol name in the ELF file to look up in the format
527 _binman_<entry>_prop_<property> where <entry> is the name of
528 the entry and <property> is the property to find (e.g.
529 _binman_u_boot_prop_offset). As a special case, you can append
530 _any to <entry> to have it search for any matching entry. E.g.
531 _binman_u_boot_any_prop_offset will match entries called u-boot,
532 u-boot-img and u-boot-nodtb)
533 optional: True if the symbol is optional. If False this function
534 will raise if the symbol is not found
535 msg: Message to display if an error occurs
536 base_addr: Base address of image. This is added to the returned
537 image_pos in most cases so that the returned position indicates
538 where the targetted entry/binary has actually been loaded. But
539 if end-at-4gb is used, this is not done, since the binary is
540 already assumed to be linked to the ROM position and using
541 execute-in-place (XIP).
544 Value that should be assigned to that symbol, or None if it was
545 optional and not found
548 ValueError if the symbol is invalid or not found, or references a
549 property which is not supported
551 m = re.match(r'^_binman_(\w+)_prop_(\w+)$', sym_name)
553 raise ValueError("%s: Symbol '%s' has invalid format" %
555 entry_name, prop_name = m.groups()
556 entry_name = entry_name.replace('_', '-')
558 entries = self._entries
559 entry = entries.get(entry_name)
561 if entry_name.endswith('-any'):
562 root = entry_name[:-4]
564 if name.startswith(root):
565 rest = name[len(root):]
566 if rest in ['', '-img', '-nodtb']:
567 entry = entries[name]
569 err = ("%s: Entry '%s' not found in list (%s)" %
570 (msg, entry_name, ','.join(entries.keys())))
572 print('Warning: %s' % err, file=sys.stderr)
574 raise ValueError(err)
575 if prop_name == 'offset':
577 elif prop_name == 'image_pos':
578 value = entry.image_pos
579 if not self.GetImage()._end_4gb:
582 if prop_name == 'size':
585 raise ValueError("%s: No such property '%s'" % (msg, prop_name))
587 def GetRootSkipAtStart(self):
588 """Get the skip-at-start value for the top-level section
590 This is used to find out the starting offset for root section that
591 contains this section. If this is a top-level section then it returns
592 the skip-at-start offset for this section.
594 This is used to get the absolute position of section within the image.
597 Integer skip-at-start value for the root section containing this
601 return self.section.GetRootSkipAtStart()
602 return self._skip_at_start
604 def GetStartOffset(self):
605 """Get the start offset for this section
608 The first available offset in this section (typically 0)
610 return self._skip_at_start
612 def GetImageSize(self):
613 """Get the size of the image containing this section
616 Image size as an integer number of bytes, which may be None if the
617 image size is dynamic and its sections have not yet been packed
619 return self.GetImage().size
621 def FindEntryType(self, etype):
622 """Find an entry type in the section
625 etype: Entry type to find
627 entry matching that type, or None if not found
629 for entry in self._entries.values():
630 if entry.etype == etype:
634 def GetEntryContents(self, skip_entry=None):
635 """Call ObtainContents() for each entry in the section
637 def _CheckDone(entry):
638 if entry != skip_entry:
639 if not entry.ObtainContents():
640 next_todo.append(entry)
643 todo = self._entries.values()
644 for passnum in range(3):
645 threads = state.GetThreads()
652 with concurrent.futures.ThreadPoolExecutor(
653 max_workers=threads) as executor:
655 entry: executor.submit(_CheckDone, entry)
658 if self.GetImage().test_section_timeout:
660 done, not_done = concurrent.futures.wait(
661 future_to_data.values(), timeout=timeout)
662 # Make sure we check the result, so any exceptions are
663 # generated. Check the results in entry order, since tests
664 # may expect earlier entries to fail first.
666 job = future_to_data[entry]
669 self.Raise('Timed out obtaining contents')
676 self.Raise('Internal error: Could not complete processing of contents: remaining %s' %
680 def _SetEntryOffsetSize(self, name, offset, size):
681 """Set the offset and size of an entry
684 name: Entry name to update
685 offset: New offset, or None to leave alone
686 size: New size, or None to leave alone
688 entry = self._entries.get(name)
690 self._Raise("Unable to set offset/size for unknown entry '%s'" %
692 entry.SetOffsetSize(self._skip_at_start + offset if offset is not None
695 def GetEntryOffsets(self):
696 """Handle entries that want to set the offset/size of other entries
698 This calls each entry's GetOffsets() method. If it returns a list
699 of entries to update, it updates them.
701 for entry in self._entries.values():
702 offset_dict = entry.GetOffsets()
703 for name, info in offset_dict.items():
704 self._SetEntryOffsetSize(name, *info)
707 contents_size = len(self.data)
711 data = self.GetPaddedData(self.data)
713 size = tools.align(size, self.align_size)
715 if self.size and contents_size > self.size:
716 self._Raise("contents size %#x (%d) exceeds section size %#x (%d)" %
717 (contents_size, contents_size, self.size, self.size))
720 if self.size != tools.align(self.size, self.align_size):
721 self._Raise("Size %#x (%d) does not match align-size %#x (%d)" %
722 (self.size, self.size, self.align_size,
726 def ListEntries(self, entries, indent):
727 """List the files in the section"""
728 Entry.AddEntryInfo(entries, indent, self.name, self.etype, self.size,
729 self.image_pos, None, self.offset, self)
730 for entry in self._entries.values():
731 entry.ListEntries(entries, indent + 1)
733 def LoadData(self, decomp=True):
734 for entry in self._entries.values():
735 entry.LoadData(decomp)
736 self.Detail('Loaded data')
739 """Get the image containing this section
741 Note that a top-level section is actually an Image, so this function may
745 Image object containing this section
749 return self.section.GetImage()
752 """Check if the entries in this section will be sorted
755 True if to be sorted, False if entries will be left in the order
756 they appear in the device tree
760 def ReadData(self, decomp=True, alt_format=None):
761 tout.info("ReadData path='%s'" % self.GetPath())
762 parent_data = self.section.ReadData(True, alt_format)
763 offset = self.offset - self.section._skip_at_start
764 data = parent_data[offset:offset + self.size]
766 '%s: Reading data from offset %#x-%#x (real %#x), size %#x, got %#x' %
767 (self.GetPath(), self.offset, self.offset + self.size, offset,
768 self.size, len(data)))
771 def ReadChildData(self, child, decomp=True, alt_format=None):
772 tout.debug(f"ReadChildData for child '{child.GetPath()}'")
773 parent_data = self.ReadData(True, alt_format)
774 offset = child.offset - self._skip_at_start
775 tout.debug("Extract for child '%s': offset %#x, skip_at_start %#x, result %#x" %
776 (child.GetPath(), child.offset, self._skip_at_start, offset))
777 data = parent_data[offset:offset + child.size]
780 data = comp_util.decompress(indata, child.compress)
781 if child.uncomp_size:
782 tout.info("%s: Decompressing data size %#x with algo '%s' to data size %#x" %
783 (child.GetPath(), len(indata), child.compress,
786 new_data = child.GetAltFormat(data, alt_format)
787 if new_data is not None:
791 def WriteData(self, data, decomp=True):
792 self.Raise("Replacing sections is not implemented yet")
794 def WriteChildData(self, child):
797 def SetAllowMissing(self, allow_missing):
798 """Set whether a section allows missing external blobs
801 allow_missing: True if allowed, False if not allowed
803 self.allow_missing = allow_missing
804 for entry in self._entries.values():
805 entry.SetAllowMissing(allow_missing)
807 def SetAllowFakeBlob(self, allow_fake):
808 """Set whether a section allows to create a fake blob
811 allow_fake_blob: True if allowed, False if not allowed
813 super().SetAllowFakeBlob(allow_fake)
814 for entry in self._entries.values():
815 entry.SetAllowFakeBlob(allow_fake)
817 def CheckMissing(self, missing_list):
818 """Check if any entries in this section have missing external blobs
820 If there are missing blobs, the entries are added to the list
823 missing_list: List of Entry objects to be added to
825 for entry in self._entries.values():
826 entry.CheckMissing(missing_list)
828 def CheckFakedBlobs(self, faked_blobs_list):
829 """Check if any entries in this section have faked external blobs
831 If there are faked blobs, the entries are added to the list
834 fake_blobs_list: List of Entry objects to be added to
836 for entry in self._entries.values():
837 entry.CheckFakedBlobs(faked_blobs_list)
839 def check_missing_bintools(self, missing_list):
840 """Check if any entries in this section have missing bintools
842 If there are missing bintools, these are added to the list
845 missing_list: List of Bintool objects to be added to
847 super().check_missing_bintools(missing_list)
848 for entry in self._entries.values():
849 entry.check_missing_bintools(missing_list)
851 def _CollectEntries(self, entries, entries_by_name, add_entry):
852 """Collect all the entries in an section
854 This builds up a dict of entries in this section and all subsections.
855 Entries are indexed by path and by name.
857 Since all paths are unique, entries will not have any conflicts. However
858 entries_by_name make have conflicts if two entries have the same name
859 (e.g. with different parent sections). In this case, an entry at a
860 higher level in the hierarchy will win over a lower-level entry.
863 entries: dict to put entries:
866 entries_by_name: dict to put entries
869 add_entry: Entry to add
871 entries[add_entry.GetPath()] = add_entry
872 to_add = add_entry.GetEntries()
874 for entry in to_add.values():
875 entries[entry.GetPath()] = entry
876 for entry in to_add.values():
877 self._CollectEntries(entries, entries_by_name, entry)
878 entries_by_name[add_entry.name] = add_entry
880 def MissingArgs(self, entry, missing):
881 """Report a missing argument, if enabled
883 For entries which require arguments, this reports an error if some are
884 missing. If missing entries are being ignored (e.g. because we read the
885 entry from an image rather than creating it), this function does
889 entry (Entry): Entry to raise the error on
890 missing (list of str): List of missing properties / entry args, each
893 if not self._ignore_missing:
894 missing = ', '.join(missing)
895 entry.Raise(f'Missing required properties/entry args: {missing}')
897 def CheckAltFormats(self, alt_formats):
898 for entry in self._entries.values():
899 entry.CheckAltFormats(alt_formats)
901 def AddBintools(self, btools):
902 super().AddBintools(btools)
903 for entry in self._entries.values():
904 entry.AddBintools(btools)