1 SWIG-1.3.12, SWIG 2.0, and Beyond
2 =================================
4 With the release of SWIG-1.3.12, I thought I'd take a few moments of
5 everyone's time to talk about the past, the present, and the future of
6 SWIG development. I'm really quite excited about the current release
7 because I think it represents a huge turning point in SWIG's
8 development. Furthermore, it is only the beginning of bigger and
9 better things to come. However, we definitely need your help.
11 To put a little perspective on the discussion, I'd start with a few
12 development statistics. In the last 12 months, there have been over
13 300 entries added to the CHANGES log and over 4000 CVS commits.
14 Although that may not sound like a lot compared to a huge software
15 project, it is significant in the context of SWIG. As a point of
16 comparison, there has been more SWIG development this year than in any
17 other year of the project and more than in the previous three years
18 combined. This even includes the first few years of development in
19 which there was also a lot of activity. Furthermore, many of the
20 recent changes have been extremely non-trivial (e.g., templates,
21 namespaces, type system, operators, etc.). As a result, SWIG is more
22 capable than I ever imagined possible.
24 Regrettably, I must admit that I've been a little negligent in
25 discussing the roadmap for where I thought this flurry of SWIG
26 development was actually headed. In part, this is because I've been
27 buried in work. However, the real reason is that I didn't really know
28 where we were going---except that in a time far far far away in the
29 future, we might arrive at some kind of "stable" release with a
30 version number other than "1.3.x". Needless to say, that's not a very
33 That said, I've spent a lot of time thinking about SWIG and trying to
34 wrap my brain around it. Specifically, just what is (or should be)
35 the primary focus of this project and what are we really trying to do?
36 That's what the rest of this message is about.
40 The first version of SWIG was written in 1995. The original system
41 was developed to help with some software wrapping problems I
42 encountered while writing molecular dynamics software at Los
43 Alamos. Later that year, I became interested in extending the wrapper
44 generator to support other scripting languages so it was rewritten in
45 C++ and modified with multiple backends (Tcl, Perl, and Guile). This
46 led to a first public release in February, 1996. Feedback from this
47 release led to a series of enhancements and the release of SWIG 1.0 in
48 September 1996. Throughout this process, my intent was to create a
49 tool that I would want to use---I never viewed the project as an
50 CS experiment in programming languages or software engineering.
54 SWIG-1.1 (June, 1997) represented a series of enhancements that were
55 added in response to feedback at conferences and from users. Shadow
56 classes, exception handling, typemaps, and a number of more useful
57 features were added. However, the overall structure of the system was
58 relatively unchanged from the initial version. Following the release
59 of 1.1, a series of minor patch releases were made. This resulted in
60 the release of SWIG-1.1p5 in February, 1998. Unfortunately, this
61 release would remain the last "stable" release for quite a long
62 time---in fact, it is still listed as the last "stable" release on the
67 Even during the development of SWIG-1.1, it was clear that the whole
68 design of the system was deeply flawed. The implementation was a mess
69 and the C/C++ support was full of holes and nasty corner cases.
70 Furthermore, there was no unifying principle that tied all of the
71 different SWIG directives together. Not only that, fixing these
72 problems appeared to be nothing short of impossible---requiring a
73 total ground-up rewrite at best. The only redeeming quality was that
74 the system basically worked "well enough," it was extensively
75 documented, and its flaws mostly known. People could use it and there
76 were work-arounds for most common problems.
78 To deal with the design problem, there were at least four attempts to
79 completely rewrite SWIG, some of which were attempted in parallel with
80 the work on SWIG-1.1. Unfortunately, none of these were ever
81 completed. The primary problem was a strong "second system" effect and
82 a desire to make SWIG do everything that one might conceivably want to
83 do with a wrapper generator (somehow). Clearly, this was a recipe for
84 disaster. In fact, all such attempts to rewrite SWIG were eventually
85 abandoned several years ago. In hindsight, I think the real problem
86 was that these rewrite efforts focused far too much attention on
87 implementation technique rather than principles. In short, the
88 failure of these efforts was due to a lack of clarity in understanding
89 how SWIG ought to work (regardless of how it was actually
92 SWIG Restart (1.3a1-1.3a5)
93 --------------------------
94 Having languished for several years, the SWIG1.1p5 release had a
95 growing pile of maintenance issues. It didn't work for newer versions
96 of certain language modules and a lot of minor bug reports and feature
97 requests had been building up. With a lot of help from Loic Dachary and
98 Thien-Thi Nguyen, we put together the 1.3a1 release (February,
99 2000). This was mostly a bug fix release to 1.1p5 except that the
100 preprocessor module from SWIG1.2 was added and a lot of minor
101 enhancements were added.
103 For the next six months, a massive effort was made to rewrite all of
104 SWIG's internal data structures (strings, lists, hashes, etc.). This
105 work was all going on underneath the covers while we tried to keep
106 SWIG in an operational state. The primary focus of this work was
107 really one of cleanup. Having given up on a total rewrite, perhaps
108 we could settle with making the implementation incrementally better
109 than before. In addition this, Matthias Koppe jumped on board to
110 reimplement the Guile module and to make other improvements to the system.
112 An important aspect of these releases was that many parts of the
113 system not directly related to wrapper code generation were removed.
114 This included the documentation system and Objective-C support. These
115 were not removed because they weren't useful. Instead, the
116 documentation system was removed because it accounted for nearly half
117 of the special SWIG directives, yet had no direct bearing on what SWIG
118 actually did. Obective-C support was removed because it was tangled
119 with C++ support in a way that was very difficult to understand and
120 maintain. The removal of these features was seen as a way to vastly
121 simplify cleanup--and to buy some time where we could rethink their
122 role in a future release.
124 SWIG Redux (1.3.6-1.3.11)
125 -------------------------
126 This work, started in February 2001, is the beginning of the current
127 SWIG implementation. With the help of William Fulton, Matthias Koppe,
128 Lyle Johnson, Luigi Ballabio, Jason Stewart, Richard Palmer, Sam
129 Liddicot, and others, this work can best be described as the wholesale
130 destruction of everything remaining from SWIG-1.1. The language
131 module API, type system, the parser, numerous SWIG directives, and
132 SWIG library files---all destroyed or rewritten. Not only that, we
133 started introducing significant incompatibilities with
134 SWIG-1.1---mostly in an effort to correct past wrongs and get
135 ourselves out of the tangled mess of earlier versions. A huge number
136 of long-standing bugs and difficult feature requests have also been
139 The general goal of this development could best be described as an
140 attempt to reduce SWIG to an easily described set of general "ideas"
141 about how it should operate. Special SWIG directives have been
142 eliminated or combined with others. Different parts of the system have
143 been better integrated. Features not directly related to wrapper code
144 generation have been removed and the system has become more
145 focused. Changes in internal data structures and APIs have allowed
146 SWIG to capture more information from interface files and to resolve
147 hard corner cases. More often than not, these are things that you
148 never notice unless you are an old user and you suddenly realize that
149 a problem you had several years back has disappeared.
151 Along with the destruction of old code, this work has quietly
152 introduced a new core--the most significant features of which are a
153 new C++ type system and multi-pass compilation. More importantly,
154 this work has really tried to provide a more principled foundation for
155 future SWIG development. However, just what is this "more principled
160 With the upcoming 1.3.12 release, SWIG is about to take a big leap
161 forward. Almost all of this is due to one realization---that almost
162 every hard problem in past SWIG releases has been directly related to
163 errors and limitations in its type system. Types are the key to
164 understanding the structure of C and C++ code. They are at the heart
165 of understanding advanced language features like namespaces, nested
166 classes, and templates. They are directly related to the data
167 marshalling that occurs in wrappers. Not only that, they interact
168 with nearly every SWIG directive. A proper type system *is* the
169 necessary foundation for moving SWIG forward.
171 To be honest, I don't think that the emphasis on types is entirely
172 obvious. In fact, a great deal of work in past SWIG rewrites has
173 focused on the problem of C++ parsing. For instance, modifying the
174 parser to handle more advanced C++ code or representing parse trees as
175 XML. Furthermore, if one looks at the SWIG mailing list, you can find
176 a *lot* of messages related to issues of C++ parsing whereas almost no
177 one ever talks about types (well, other than typemaps). Even other
178 wrapper generation tools seems to spend a lot of time dealing with the
179 parsing issue. Although parsing is clearly important, I don't think it
180 has ever been the real problem in SWIG. This is because even though a
181 parser can tell you what's in a header file, it doesn't tell you
182 anything about how the different pieces of the system behave or how
183 they might interact. To do that, you need to do a lot more than just
184 parsing--and that's really the whole point.
186 Although earlier 1.3 releases have made big improvements to the type
187 system, SWIG-1.3.12 is the first release that really tries to tackle
188 the type-system issue in a major way. We have patched nearly all
189 remaining holes in the type system and we have added full support for
190 C++ namespaces. Not only that, we have completely reimplemented C++
191 template support in a way that supports templates, member templates,
192 and template specialization. Luigi and I are currently using this to
193 work on proper SWIG library support for parts of the C++ standard
194 library and the Standard Template Library (STL). Although some crusty
195 C programmers (present company excepted), might balk at such apparent
196 insanity, this work has impacted all parts of SWIG at all levels.
197 Even a variety of subtle errors in C support have been fixed by this
200 In addition to the type system work, SWIG-1.3.12 contains continued
201 reduction in the implementation. Directives have been removed,
202 refined, renamed, or consolidated. We're still narrowing the focus of
203 the system and working towards some kind of convergence. "Convergence
204 to what?!?", you ask.
208 In a nutshell, SWIG is a C/C++ declaration compiler that generates
209 wrapper code (okay, so you knew that much). However, to really
210 understand what SWIG is doing and where SWIG-1.3.x is headed, it is
211 useful to know that the whole system is essentially being built around
212 three extensions to the C++ type system:
214 - Typemaps. Typemaps are rules that define the process by which
215 data is converted between languages. They are fully integrated
216 with the C++ type system and they are applied using a type-based
217 pattern matching mechanism. All data conversion SWIG is
218 defined by typemaps and is fully reconfigurable.
220 - Declaration annotation. There are special directives that modify
221 the wrapping behavior of individual declarations. Declarations
222 can be selectively identified and decorated with arbitrary
223 attributes that affect wrapper generation. Like typemaps,
224 declaration matching is fully integrated with the C++ type system.
225 Almost all SWIG customization directives are a form of declaration
228 - Class extension. The ability to extend classes and structures
229 with new methods/attributes when building wrappers. Classes
230 are part of the type system so class extension is naturally
231 integrated with the C++ type system as well (big surprise).
233 And that's it--this is the end-game of SWIG-1.3.x development. When
234 stabilized and documented it will become SWIG-2.0.
238 I really want to emphasize that all of this work is part of a much
239 bigger picture. SWIG is used by a surprising number of people in
240 industry, government, and academia. It's used to build systems used
241 by millions of people every day. It has even shown up in video games
242 and other unlikely settings. Although SWIG doesn't have the same
243 visibility as many large software projects, over 12000 people have
244 downloaded SWIG-1.3.11 in the last 4 months. Clearly someone is using
245 it for something! Because of this, I think it is important for us to
246 work on moving SWIG towards a more solid foundation. Doing so will
247 give the system more credibility and long term viability---and it will
248 be a lot more fun to use!
250 It's also worth noting that there are some rather interesting CS
251 connections at work here. Extensions to the type system and typemaps
252 have some interesting relations to work in programming languages. The
253 SWIG declaration annotation system and class extension feature seem
254 oddly similar to work in the emerging area of Aspect Oriented
255 Programming (AOP). There are undoubtedly connections to other areas
256 of software engineering and architecture.
258 The key point is that SWIG isn't going to connect to anything if
259 no-one can quite describe what it is or how it works.
261 SWIG-2.0 and the Future
262 -----------------------
263 SWIG-1.3.12 still represents work in progress. There are bugs, the
264 documentation is still incomplete, and there are parts of the
265 implementation that are rather ugly. We are also still working out a
266 few very hard problems like nested classes, callback functions, and
267 overloading. A few old features are still missing (Objective-C,
268 documentation). However, I believe the end of the 1.3.x series is
271 Over the summer, a few more 1.3.x releases may appear but the current
272 plan is to aim for a SWIG-2.0 release in September. This release is
273 really moving towards the design principles described above and will
274 be a very major enhancement over SWIG-1.1.
276 As for the future, a number of interesting ideas are on the table. I
277 want to add support for contracts/assertions in order to solve some
278 reliability issues that arise when retrofitting legacy codes with a
279 scripting interface. Support for an extension language has been
280 promoted by David Fletcher and was suggested by someone else on the
281 mailing list rather recently. I have a graduate student working on
282 SWIG support for the Microsoft CLR and .NET languages. Other work
283 might include support for alternative parsers, dynamically loadable
284 language modules, and so forth. However, none of this is really going
285 to materialize if we can't get the 2.0 release stablized. In fact, I
286 see SWIG-2.0 as a necessary step for moving forward with these ideas.
288 We need your help! Yes, you.
289 ----------------------------
290 Nobody gets paid to work on SWIG. The developers are volunteers who
291 work in their spare time. Furthermore, SWIG is not supported by
292 investors, a large corporation, or research grants. I work on it
293 because it's fun, challenging, and useful. I presume that other
294 developers do the same. However, we only have limited resources and
297 - If you like SWIG and find it useful, we need you to try new versions.
298 We want you to torture test the releases and to break them. We need
299 bug reports. No bug is too obscure or unimportant---we *will* fix it
300 if we can. We also need feedback about things that are annoying or
301 compatibility features that might help in going from 1.1 to 2.0.
303 - We need help with documentation, examples, testing, libraries, and all
304 sorts of other aspects of the release. Even if you have never
305 written a SWIG language module or dived into its implementation,
306 there are many ways that you can help. Consider writing a case study
307 about how you wrapped a big library. Contribute tests that break the
308 implementation in horrible ways. Help with the web page or FAQ.
310 - Most of the SWIG-1.3.x work has focused on the SWIG core. However, as
311 the 2.0 release nears, we will be working on a variety of enhancements
312 to the language modules. If there are things you would like to see
313 in any of the language modules, you need to let us know.
315 - There are SWIG language modules that have not made it into the
316 distribution. Examples that I know about include ITCL, Swig-Eiffel,
317 and Swig-Lua. We would gladly make these part of the standard SWIG
318 distribution. However, we also need help to do it. Porting from
319 SWIG-1.1 is no easy task, but we're more than willing to help. It's
320 not as bad as one might imagine.
322 - We are always looking for developers. Subscribe to
323 the swig-dev mailing list, details at http://www.swig.org/mail.html,
324 or send me email to get involved.
328 I'd just like to thank everyone who has submitted feedback, bugs, made
329 contributions, and put up with my occasional thrashing over the years.
330 I welcome any comments about this document and how we can make SWIG even
333 Dave Beazley (beazley@cs.uchicago.edu)
projects / external / swig.git / blob