1 Persistent Storage System
2 ~~~~~~~~~~~~~~~~~~~~~~~~~
9 The Storage System is responsible for persistently storing and
10 loading Patches and Atoms for the packagemanager. Other object types
11 might be added in future.
17 Required operations are:
19 - read all objects of a given type
20 - read in an object identified by a given attribute
23 - delete an object (only for cleanup after update, does not need to
25 - a query interface similar to what rpm offers
34 In order of relevance:
36 - It will be part of the ZYPP library, and as such needs to be
37 reentrant and thread safe.
39 - There is only one attribute in an object that might need to be
40 modified, the status. Thus, no universal "modify object" operation is
43 - Most time critical operation is "read all objects", since this will
44 defer the startup of the YaST packagemanager. This already takes an
45 uncomfortably long time.
47 - It should try to cut down on memory usage, since it will be in use at
48 install time, at least when updating.
50 - We'll probably get something in the order of 100-1000 of these objects.
52 - It should be possible to use different backends for the actual low
53 level storage, i.e. Berkley db, postgres, mysql, flat file.
60 Here's a layer diagram of the main architecture. The paranthesed
61 figures indicate APIs::
63 +--------------------+--------------+
64 | caller, e.g. package manager |
65 +--------(1)---------+ +
67 +-------------------(2)-------------+
68 | Persistent Storage Core |
69 +--------(3)---------+------(4)-----+
70 | Backend plugin | Type plugin |
71 +--------------------+--------------+
73 | (e.g., Berkley DB) | |
74 +--------------------+ +
76 +--------------------+--------------+
91 These components are described below, the APIs are subject to a later,
92 more detailed follow-up.
99 This implements the rpm-like query operation by using general and
100 simpler search operations within the Core API.
103 Persistent Storage Core
104 =======================
106 The Core is the "main contact" for the layers above and breaks down
107 the complete functionality for the simpler modules in the lower layer.
113 The backend plugin considers data objects in a form that is suitable
114 for the backend, i.e. a data record is represented by:
117 - a set of keys, which are
118 - pairs (attribute name, value)
120 The following operations are available:
122 - create the storage database
123 - find a record by a given attribute and return a handle
124 - insert a new record and return a handle to it
126 These operations affect the record that is referenced by a handle:
130 - remove it (delete it)
136 These are responsible for dealing with the specifics of the type of
137 objects that are stored. This includes:
139 - knowing about the path names of the database
140 - knowing about the attributes and keys
141 - converting the internal representation of a data object to XML and back
142 - extracting the key values from an XML string
148 The parser creates simple structure-like objects from the XML string.
149 This is derived from the already implemented XMLNodeIterator class.
156 This is an overview of backends that look promising for handling the
162 Basic idea: Store the data for each object as an XML element in a flat
163 file, store the status separately and add indices for fast access.
165 For each type there is
167 - a master file, which contains:
169 - the pathnames of the other files
172 - one or more data files, which contains all the data as XML, except
175 - the status for each object as a binary file, each byte representing
176 the status for one object. The association between status byte and
177 object is contained in an index file.
179 - one or more index files (for all data fields that need to be a
180 key). The index could either be an ordered flat file to mmap, or
181 something like a tree or hash table. (I'm still looking for a suitable
184 Advantage: More direct control over everything, can be tailored to our
185 needs, probably the efficients way to do it.
191 Berkley DB is a small non-relational embedded database library that
192 provides compact transparent data storage with a wealth of related
193 services. Data is managed as (key,value) tuples, and indexing is
194 performed for the keys.
196 Basic idea: Store the data for each object as an XML string in a
197 berkley db. Fast access and everything is managed by the db. The
198 Berkley db is already used within rpm, so it's no additional
199 dependency. The library is 1-2MB (depending whether you use the C++
202 Another option: Instead of saving the XML string, serialize the
203 ParserData structure and save this one directly. Faster, but more
204 difficult to access for debugging.
206 Advantage: Everything's there.
208 Disadvantage: Berkley DB had some issues in the past time. We'll
211 Side note: xmldb cannot be used since it needs a lot of special libraries with
212 specific versions. For this reason, it hasn't made it on the SUSE
221 SQLite is an embedded and very small SQL engine that contains both
222 server and client, and has no multi-user access support. It is very
223 small (200-300 KByte).
225 Basic idea: Store the data as an XML string in a record. For each key
226 data field, add another attribute.
228 Disadvantage: We don't really need the relational aspects if SQLite,
229 and we don't have many people with knowledge of this topic in the
230 theme. Not much experience with it. Additional library in the install
231 system. Probably worst performance (but SQLite claims being fast for SQL)
233 Advantage: Maximum flexibility, especially in queries. Easy changes in