tag:blogger.com,1999:blog-55687888827600663202024-03-13T07:20:47.789+01:00lichtblau's blogDavid Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.comBlogger22125tag:blogger.com,1999:blog-5568788882760066320.post-71388166661596622282011-06-19T19:47:00.001+02:002011-06-19T19:48:17.868+02:00Parsing and serializing with SAX<p>
Several years ago, I started working on the XML parser that Gilbert
Baumann had written as part of his web browser called
"<a href="http://common-lisp.net/project/closure/">Closure</a>".
Since then, "Closure XML"
(or <a href="http://common-lisp.net/project/cxml/">cxml</a> for short)
has developed into a set of little libraries, one main goal being
completeness and correctness with regard to the various standards they
are following.
</p>
<p>
And standards abound in XML land, which is nice for implementors
(thanks to the good test suites!) and nice for users (because the
specs partially serve as documentation, and make it easy to transition
between different languages implementing them). But I've always tried
to release cxml with enough documentation to get users started for all
the parts that are implementation-specific. And not all areas are
covered by standards: Of course, the document format itself is
specified strictly; the same goes for XPath, XSLT, schemas, etc.
</p>
<p>
But little is standardized in terms of API support, and that sort of
choice is generally good: After all, a Lisp XML parser should fit into
the Lisp world and not mimick (say) JavaScript too much. But many
good ideas can be borrowed from other languages. Examples inspired by
Java are
<a href="http://www.lichteblau.com/cxml-stp/">STP</a>, motivated
heavily by <a href="http://www.xom.nu/">XOM</a> (and tweaked for added
lispiness) -- and SAX:
</p>
<p>
<a href="http://en.wikipedia.org/wiki/Simple_API_for_XML">SAX</a> is a
classic Java API. It defines a protocol of methods that get called by
an XML parser, and each method call signifies an <i>event</i>
(e.g. that the parser saw a XML tag). In cxml, SAX is one of two
fundamental APIs offered (the other being a StAX-like pull-based
interface), and it's essential to its inner workings. Yet I had never
bothered to document it fully. For one thing, everyone seemed to know
SAX from Java anyway. It's also hidden from view for most users. And
ultimately, it's just a list of generic functions, right?
</p>
<p>
Technically it is just that, and yet it's central to communication
between cxml's libraries, and it makes parsing and serialization in
cxml <i>modular</i> and <i>reusable</i>.
Hence <a href="http://xach.livejournal.com/">some user</a>s had long
suggested to me that I should explain SAX in full.
</p>
<p>
So here it is:
<a style="font-weight: bold" href="http://common-lisp.net/project/cxml/saxoverview/">The SAX
overview</a>.
</p>
<p>
TL;DR: Skip to the link above.
</p>David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com1tag:blogger.com,1999:blog-5568788882760066320.post-44495187051136075292011-06-18T21:28:00.002+02:002011-06-18T21:42:13.418+02:00SBCL on github and gitorious<p>SBCL <a href="http://random-state.net/log/3516349972.html">moved to git</a> last week, and with the main repository hosted on <a href="http://sbcl.git.sf.net">sf.net</a>, several users have asked for an SBCL mirror on github.
<p>Automatically synced mirrors are available on github and gitorious (where SBCL branches had, of course, also been uploaded by contributors long before the official switch to git), and it will now be easier than ever to make your own repository and branches.
<p>The mirrors update several times a day:
<p>
<ul>
<li><a href="http://github.com/sbcl/sbcl">github.com/sbcl</a>
<li><a href="http://gitorious.org/sbcl/sbcl">gitorious.org/sbcl</a>
</ul>
<p>Please test, and good luck with your SBCL hacking efforts!David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com0tag:blogger.com,1999:blog-5568788882760066320.post-75616774613176145252010-04-07T09:00:00.002+02:002010-04-07T14:20:51.740+02:00More loopingFor context, see Nathan's blog.
[<b>Edit:</b> I've just learned on IRC from Maciej Katafiasz that iterate also has a <a href="http://common-lisp.net/project/iterate/doc/Named-Blocks.html#Named-Blocks">feature</a> built in which renders this hack unnecessary. Even more iterate awesomeness! I'll leave this post up though, because I still think it' s pretty cute that SBCL optimizes this case so nicely.]
<pre>
MY-USER> (defun outer-collect-in-inner-loop ()
(declare (optimize speed (safety 0) (debug 0)))
(iter (for outer from 0 below 10)
(declare (type (integer -1 10) outer))
(flet ((outer-collect (x) (collect x)))
(iter (for inner from 0 below 10)
(declare (type (integer -1 10) inner))
(outer-collect (+ (* 10 outer) inner))))))
OUTER-COLLECT-IN-INNER-LOOP
</pre>
Note that the COLLECT in the outer loop is ITERATE's collecting feature. The trick is to use it in a FLET before starting the inner loop, thereby making it available further down.
Declarations added only to get a nicer disassembly.
<pre>
MY-USER> (OUTER-COLLECT-IN-INNER-LOOP)
(0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99)
</pre>
Okay, it works. But how is it being compiled?
<pre>
MY-USER> (disassemble 'OUTER-COLLECT-IN-INNER-LOOP)
; disassembly for OUTER-COLLECT-IN-INNER-LOOP
; 069222FF: 31C9 XOR ECX, ECX ; no-arg-parsing entry point
; 301: BE17001020 MOV ESI, 537919511
; 306: BF17001020 MOV EDI, 537919511
; 30B: BA17001020 MOV EDX, 537919511
; 310: 48C7C1F8FFFFFF MOV RCX, -8
; 317: E977000000 JMP L5
; 31C: 90 NOP
; 31D: 90 NOP
; 31E: 90 NOP
; 31F: 90 NOP
; 320: L0: 31DB XOR EBX, EBX
; 322: 48C7C3F8FFFFFF MOV RBX, -8
; 329: EB5B JMP L4
; 32B: 90 NOP
; 32C: 90 NOP
; 32D: 90 NOP
; 32E: 90 NOP
; 32F: 90 NOP
; 330: L1: 488BD1 MOV RDX, RCX
; 333: 486BD20A IMUL RDX, RDX, 10
; 337: 4C8BCB MOV R9, RBX
; 33A: 4E8D040A LEA R8, [RDX+R9]
; 33E: 4989AC24B8000000 MOV [R12+184], RBP
; 346: 4D8B5C2468 MOV R11, [R12+104]
; 34B: 498D5310 LEA RDX, [R11+16]
; 34F: 4939542470 CMP [R12+112], RDX
; 354: 765C JBE L7
; 356: 4989542468 MOV [R12+104], RDX
; 35B: 498D5307 LEA RDX, [R11+7]
; 35F: L2: 4931AC24B8000000 XOR [R12+184], RBP
; 367: 7402 JEQ L3
; 369: CC09 BREAK 9 ; pending interrupt trap
; 36B: L3: 4C8942F9 MOV [RDX-7], R8
; 36F: 48C7420117001020 MOV QWORD PTR [RDX+1], 537919511
; 377: 4881FE17001020 CMP RSI, 537919511
; 37E: 7529 JNE L6
; 380: 488BF2 MOV RSI, RDX
; 383: 488BFA MOV RDI, RDX
; 386: L4: 4883C308 ADD RBX, 8
; 38A: 488BD3 MOV RDX, RBX
; 38D: 4883FA50 CMP RDX, 80
; 391: 7C9D JL L1
; 393: L5: 4883C108 ADD RCX, 8
; 397: 488BD1 MOV RDX, RCX
; 39A: 4883FA50 CMP RDX, 80
; 39E: 7C80 JL L0
; 3A0: 488BD6 MOV RDX, RSI
; 3A3: 488BE5 MOV RSP, RBP
; 3A6: F8 CLC
; 3A7: 5D POP RBP
; 3A8: C3 RET
; 3A9: L6: 48895701 MOV [RDI+1], RDX
; 3AD: 488BFA MOV RDI, RDX
; 3B0: EBD4 JMP L4
; 3B2: L7: 6A10 PUSH 16
; 3B4: 4C8D1C25601B4200 LEA R11, [#x421B60] ; alloc_tramp
; 3BC: 41FFD3 CALL R11
; 3BF: 5A POP RDX
; 3C0: 488D5207 LEA RDX, [RDX+7]
; 3C4: EB99 JMP L2
</pre>David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com2tag:blogger.com,1999:blog-5568788882760066320.post-40872260493632383412009-08-12T13:23:00.007+02:002009-08-12T18:32:28.107+02:00cl-perec blog series by pinterface<p>For several years, the best (and only) option for portable access to relational databases from Common Lisp has been the CLSQL library, a free reimplementation of LispWorks' CommonSQL interface with support for many backend databases, a lispy SQL abstraction as well as simple object-relational mapping support.
<p>Recently, several alternatives have emerged: Postmodern provides particularly good support tailored to PostgreSQL only, other libraries abstract away from relational databases entirely, and finally, there are two new stars on the library horizon: CL-RDBMS and CL-PEREC.
<p>Having used CLSQL for several years, I am currently investigating a switch to CL-RDBMS and PEREC for a code base of non-trivial size, aiming to improve scalability, portability across databases, and last not least readability of the code base.
<p>And I am very happy with everything I have found so far:
<p>In the CL-RDBMS/PEREC ecosystem, work is split into two layers.
<ul>
<li>
CL-RDBMS forms the lower layer. It abstracts over database access libraries (currently with support for PostgreSQL, Oracle, and SQLite) and defines an optional, Lispy, extensible syntax for SQL for data definition and query statements.
<li>The optional PEREC layer is the one which really shines. It is an object-relational mapping (ORM) seamlessly integrating CLOS classes with database tables.
</ul>
<p>Highlights:
<ul>
<li>Sensible caching model: Configurable caching of slots within a transaction, but not between transactions.
<li>Object identity preserved (within a transaction).
<li>Protection against accidental cross-transaction access.
<li>Lispy SELECT statements using code that looks like ordinary CLOS use, but which usually compiles down to efficient database-side SQL statements.
</ul>
<p>Overall, an architecture that is ideal for robust, scalable setups with multiple threads and processes without risk of the "oversold concert tickets" situation reported for a popular Lisp web site based on older ORM software a few years ago...
<p>The only fly in the ointment for me was the lack of introductory material. PEREC comes with an extensive test suite and nice examples, but I would have wished for a few more hints to get me started.
<p>Thanks to fellow blogger <a href="http://pinterface.livejournal.com/">pinterface</a>, my wish has been granted. He has written a blog series about PEREC with many details about getting set up, and including various customization tips.
<p>Articles <a href="http://pinterface.livejournal.com/tag/cl-perec">in the series</a>:
<ol>
<li><a href="http://pinterface.livejournal.com/34706.html">Getting Started with cl-perec</a>
<li><a href="http://pinterface.livejournal.com/35042.html">Persisting Simple Types with cl-perec</a>
<li><a href="http://pinterface.livejournal.com/35586.html">Sensible Serializing with cl-perec</a>
<li><a href="http://pinterface.livejournal.com/35935.html">Peering Down the Rabbit Hole with cl-perec</a>
</ol>
<p>PEREC is still evolving, but the tips presented are definitely helpful for PEREC as implemented today. Also check the comments on the blog articles, written by PEREC's author!David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com2tag:blogger.com,1999:blog-5568788882760066320.post-85281038091141513552009-04-14T08:37:00.007+02:002009-04-14T12:32:47.260+02:00Ben Hyde's project dependency graphBen Hyde <a href="http://enthusiasm.cozy.org/archives/2009/04/the-briar-patch">writes about clbuild</a> and has a very nice graph of project dependencies in his post.<br>
<br>
His approach is extraordinarily simple and uses a small Lisp program to parse clbuild's dependencies file and write out a .dot file. Of course, using Graphviz is a well-known approach for this kind of dependency diagram. Bill Clementson <a href="http://bc.tech.coop/blog/051012.html">mentioned</a> Richard Newman's graphs a few years ago.<br>
<br>
But Ben's graph is a fresh take on this because of the way it uses clbuild's recorded dependency information, which shows dependencies of <i>projects</i> rather than dependencies of <i>systems</i>. What's the difference between projects and systems? Lisp projects often contain multiple .asd files.<br>
<br>
Before thinking about the details of that distinction, let's look at the picture. Here's his graph, showing Hunchentoot, Drakma, cxml, and others:<br>
<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFeVy0Yx8-pOSHsvsT0XaEBYA1Kt_PhaBPjwxqeim9hoe8j7eXpuvwIGUQpzGI6jyMuE9T77q4P0JvjVbEB_bYimUJ0vXEezfsqpx15_N7nZB07rs1ko_P45Qu1w928oxMrVTUjkz4dAI/s1600-h/ben-deps-full.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 236px; height: 320px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFeVy0Yx8-pOSHsvsT0XaEBYA1Kt_PhaBPjwxqeim9hoe8j7eXpuvwIGUQpzGI6jyMuE9T77q4P0JvjVbEB_bYimUJ0vXEezfsqpx15_N7nZB07rs1ko_P45Qu1w928oxMrVTUjkz4dAI/s320/ben-deps-full.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5324440139370138466" /></a><br>
Observations:<br>
<ul>
<li>Yes, there's a lot of <tt>Babel</tt> in there. (It deserves it!)<br>
<li>The project dependency graph is not transitive. Note how <tt>CXML</tt> depends on <tt>Babel</tt>, which in turn needs <tt>Stefil</tt>. But <tt>CXML</tt> itself does not depend on <tt>Stefil</tt>. How can that be? It's because the <i>system</i> <tt>CXML</tt> depends only on the <i>system</i> <tt>babel</tt>. And that's not where the use of <tt>Stefil</tt> comes from. It's actually just <tt>babel-tests</tt> that needs <tt>Stefil</tt>, and that isn't needed when you're compiling <tt>CXML</tt>.<br>
<li>Where there <i>are</i> indirect dependencies, the graph shows them in full. Note that CL+SSL has an arrow to babel here, although it doesn't use it directly. This indirect dependency is due to its use of CFFI.
</ul><br>
While probably not suitable for all dependency graphs, I think that this explicit display of indirect dependencies gives very nice results in this case, because it highlights commonly used libraries like Babel more than an ordinary graph would have done.<br>
<br>
You can download his code at <a href="http://github.com/bhyde/clbuild2dot/tree/master">github</a>. Note that you don't even have to download the projects that you want to compute a graph for, if you replace the call to <tt>directory</tt> with any other test of your choice.David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com2tag:blogger.com,1999:blog-5568788882760066320.post-3623381649223005882009-04-04T11:59:00.000+02:002009-04-04T12:00:11.901+02:00XSLT 1.0 implemented in Common Lisp<p>
Newly
released: <a href="http://common-lisp.net/project/xuriella/">Xuriella
XSLT</a>, the first implementation of XSLT written entirely in Common
Lisp.
</p>
<p>
Based
on <a href="http://common-lisp.net/project/plexippus-xpath/">Plexippus
XPath</a>, Xuriella is an implementation of XSLT 1.0, written by Ivan
Shvedunov and me.
</p>
<p>
Xuriella is quite complete and correct -- we run the official
testsuite, with more than 95% of tests currently passing.
</p>
<p>
<b>Extensions</b>
</p>
<p>
One advantage of a pure-Lisp implementation is that extension
elements (as well as XPath extensions) can be defined easily.
</p>
<p>
That's a huge plus because XSLT itself is a very specialized
programming language -- it excels at XML/HTML generation and
transformation only. Being able to write custom extensions in Lisp
helps with any non-XML-ish parts of the task which XSLT itself might
not handle conveniently.
</p>
<p>
<b>Documentation</b>
</p>
<p>
If you just want to try applying stylesheets, there are only two
functions you need to know about: <tt>parse-stylesheet</tt>
and <tt>apply-stylesheet</tt>.
</p>
<p>
For details about these functions (and all others, including those for
extensions), refer to the
<a href="http://common-lisp.net/project/xuriella/atdoc/index.html">API documentation</a>.
</p>
<p>
<b>Example</b>
</p>
<p>
The <a href="http://common-lisp.net/project/xuriella/example.html">example</a>
uses Hunchentoot and Xuriella with XSLT as a template language in a
simple directory listing request.
</p>David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com4tag:blogger.com,1999:blog-5568788882760066320.post-32420247778569101752008-12-07T11:58:00.001+01:002008-12-07T12:00:05.993+01:00Plexippus XPath released<p>
The first version of Ivan
Shvedunov's <a href="http://common-lisp.net/project/plexippus-xpath/">Plexippus
XPath</a> has been released, an implementation of XPath 1.0, written
entirely in Common Lisp.
</p>
<p>
Plexippus has been available from darcs for a few months now, and we
have had some early users (thanks for testing the code at that early
stage), but this is the first tarball release. New tarballs for cxml
and its other add-on projects are also available now, so that you don't
have to juggle with git, darcs, and CVS anymore.
</p>
<p>
<b>Beta version: SBCL-only</b>
</p>
<p>
Plexippus is well-tested on SBCL, but has not yet been ported to any
other Lisp implementation. That's why this release is declared as a
beta version.
</p>
<p>
(Ports to other Lisps mainly involve support code for non-trapping
floating-point arithmetic. Patches are welcome.)
</p>
<p>
<b>The XPath protocol</b>
</p>
<p>
For extensibility, Plexippus defines a protocol of generic functions
to implement the XPath data model, so any object model for XML can be
queried using Plexippus if it implements these generic functions.
Out-of-the-box, there is support
for <a href="http://www.lichteblau.com/cxml-stp/">cxml-STP</a> (a DOM
alternative) and cxml's DOM.
</p>
<p>
<b>Some examples</b>
</p>
<p>
Let's parse a test document first (no XPath involved yet):
</p>
<pre style="background-color: #eeeeee; border: 1px solid #cccccc;">
CL-USER> (defparameter *document*
(cxml:parse "<test a='1' b='2'>
<child>hello world</child>
</test>"
(stp:make-builder)))
*DOCUMENT*
</pre>
<p>
Find and add the two attributes as numbers:
</p>
<pre style="background-color: #eeeeee; border: 1px solid #cccccc;">CL-USER> (xpath:evaluate "test/@a + test/@b" *document*)
3.0d0</pre>
<p>
Find the element called <tt>child</tt> and its string value.
(Using <tt>string()</tt> in the expression itself would also have worked.)
</p>
<pre style="background-color: #eeeeee; border: 1px solid #cccccc;">CL-USER> (xpath:string-value (xpath:evaluate "//child" *document*))
"hello world"</pre>
<p>
Iterate over all elements, in document order:
</p>
<pre style="background-color: #eeeeee; border: 1px solid #cccccc;">CL-USER> (xpath:do-node-set (node (xpath:evaluate "//*" *document*))
(format t "found element: ~A~%"
(xpath-protocol:local-name node)))
found element: test
found element: child</pre>
<p>
<a href="http://common-lisp.net/project/plexippus-xpath/examples.html">More
examples here</a>.
</p>David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com1tag:blogger.com,1999:blog-5568788882760066320.post-36110574252977935012008-07-26T18:38:00.001+02:002008-12-11T07:41:19.880+01:00lisp-berlin<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://www.katzundgoldt.de/rumpfkluft.htm"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4NI1dL6mc3xsOnN3RTbf91CZMfb9S1akqcONjJvfk1Fs77h6Av6rCFTvXBHV1AnqhJK1HliHsQyahG573H2qekTBKhFprL3qSxwnZxSqXlAjRqW43iDyrbP2R0toFDQQzLhsYSpV1ZGk/s320/toll.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5227360037167581746" /></a>[rumpfkluft.de]
<br>
<br>
Lispers are meeting up on a regular basis almost <a href="http://xach.livejournal.com/173670.html">everywhere</a>. But does Berlin have a Lisp user group? If so, where would you start looking for it? Until recently, the only resources available were the mailing lists <a href="http://common-lisp.net/mailman/listinfo/eurolisp">eurolisp</a> (targeting everyone and no one in particular) and <a href="http://common-lisp.net/mailman/listinfo/lisp-hh">lisp-hh</a> (frequented by Hamburg Lispers).
<br>
<br>
Now, thanks to <a href="http://netzhansa.blogspot.com/">Hans</a>, we have our own Wiki and mailing list. For your future Lisp needs in Berlin, we present lisp-berlin: <a href="http://groups.google.com/group/lisp-berlin">http://groups.google.com/group/lisp-berlin</a>
<br>
<br>
See <a href="http://groups.google.com/group/lisp-berlin/web/treffen">lisp-berlin/treffen</a> for Katrin's "Stammtisch", which has been a regular meeting for about a year now.
<br>
<br>
We would like to organize meetings replacing the Stammtisch later this year, ideally in a place that is not as excessively casual, so stay tuned for more.David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com0tag:blogger.com,1999:blog-5568788882760066320.post-7526534638115457132008-02-05T21:48:00.000+01:002008-02-05T21:47:56.725+01:00clbuild FAQ<p>
Since bloggers started covering clbuild (<a href="http://boinkor.net/archives/2008/01/a_public_service_announcement.html">1</a>,<a href="http://bc.tech.coop/blog/080116.html">2</a>), there has been a noticable surge of newbie questions.
</p>
<p>
To record answers to those questions, there is now a longer <a href="http://common-lisp.net/project/clbuild/#faq">FAQ</a> for clbuild, covering these topics:
</p>
<ul>
<li> How does clbuild differ from asdf-install?
<li> My favourite application is not supported. How can I
add it?
<li> Why did you get rid of all tarball-only downloads?
<li> It doesn't load my ~/.sbclrc!
<li> "clbuild lisp" doesn't seem to use my monster.core!
<li> Can I start the lisp with swank already preloaded?
<li> Can I run emacs and slime without going through "clbuild slime"?
<li>Is the "dependencies" file autogenerated?
<li>The "dependencies" file is broken!
<li>Why is clbuild written in shell? Lisp is so much better.
<li> Does it work on cygwin?
</ul>
<p>
Note the slime-related changes, which allow swank startup the way Bill is explaining it in his post, but without the workarounds.
</p>David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com2tag:blogger.com,1999:blog-5568788882760066320.post-45275206454884468422007-12-25T21:46:00.000+01:002007-12-26T00:21:52.151+01:00comp.lang.lisp needs to stop complaining about overcommit<i>or: How I got sucked back into SBCL hacking on Christmas.</i>
<pre>
: david@radon:~; ps -o pid,vsz,rss,comm -p `pidof sbcl`
PID VSZ RSS COMMAND
1019 570424 4424 sbcl
: david@radon:~; ps -o pid,vsz,rss,comm -p `pidof sbcl`
PID VSZ RSS COMMAND
1019 963320 404204 sbcl
: david@radon:~; ps -o pid,vsz,rss,comm -p `pidof sbcl`
PID VSZ RSS COMMAND
1019 3988328 825804 sbcl
</pre>David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com4tag:blogger.com,1999:blog-5568788882760066320.post-15308799949297085182007-10-27T16:27:00.000+02:002007-10-27T16:27:35.241+02:00Ten years of Closure HTMLMore than a decade ago, Gilbert Baumann started writing the Closure web browser. It includes a great HTML parser, written all in Lisp.<br>
<br>
Released today, <a href="http://common-lisp.net/project/closure/closure-html">Closure HTML</a> is a stand-alone version of the parser.<br>
<br>
It supports HTML 4, understands malformed HTML, and can (optionally) be used in conjunction with Closure XML and its data structures.<br>
<br>
An easy way to get started with Closure HTML itself is with its LHTML builder, which represents HTML elements as simple lisp lists.<br>
<br>
Together, the two parsers can be used to turn HTML into XHTML or vice versa, and in particular to parse HTML into DOM or STP. Even for users who only parse and work with XHTML internally, the new code can be useful to emit normal HTML 4 as the last step of processing.David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com0tag:blogger.com,1999:blog-5568788882760066320.post-31005955520402399302007-08-30T22:47:00.000+02:002007-08-30T22:58:21.561+02:00Forking SBCL for Dummies<p>
<a href="http://repo.or.cz/">http://repo.or.cz/</a> hosts a git repository for SBCL
and makes it easy to publish your own fork of SBCL.
</p><p>
Here is a quick step-by-step
guide for anyone planning to have his repository hosted there.
(All of this will be painfully obvious to the git experts.)
<ul>
<li><a href="http://repo.or.cz/m/reguser.cgi">Register a user account</a>. All you need is an SSH public key, no questions asked.</li>
<li>Create the fork. Find the <a href="http://repo.or.cz/w/sbcl.git">SBCL project</a> and go to "fork". Enter a project name for the fork and an admin password.</li>
</ul>
</p><p>
Done. Now you have a fork, but you need to initialize it first.
</p><p>
<ul>
<li>Go to your "Project Settings" page and <em>add yourself as a user</em>.
</li>
</ul>
</p><p>
Otherwise you cannot push to your own project.
<ul>
<li>Push into the fork. One way to do this is to clone the normal SBCL repository, then use
<pre>git push --all ssh://yourusername@repo.or.cz/srv/git/sbcl/yourprojectname.git</pre></li>
</ul>
</p><p>
Don't forget the --all, which instructs git to push all refs. Whatever a ref is, anyway.
</p>David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com3tag:blogger.com,1999:blog-5568788882760066320.post-10303876925815903402007-08-26T21:06:00.000+02:002007-08-26T22:30:32.813+02:00cloak<p>
It is unfinished, slow, buggy, unmaintained, in need of a rewrite -- and now <b>you</b> can hack it yourself!
</p>
<p>
Doesn't that sound exciting?  <i>Of course</i> it does.
</p>
<p>
<b>Prerequisites.</b> Only Linux/x86 is supported<sup>1</sup>. You will need several hours of spare CPU time, about 1 GB of RAM, and lots of disk space.
Compilation involves building <a href="http://www.sbcl.org/">SBCL</a> and <a href="http://www.gnu.org/software/classpath/">classpath</a> first, so make sure to install all required dependencies first. Debian users can run
<pre># apt-get install sbcl svn cvs wget jikes
# apt-get build-dep classpath</pre>
to do so.
</p>
<p>
<b>Build script.</b> Grab cloak using git [edit: needs git 1.5, no idea why]:
<pre>$ git clone <a href="http://www.lichteblau.com/git/cloakbuild.git">http://www.lichteblau.com/git/cloakbuild.git</a></pre>
and compile it using clbuild-like commands:
<pre>$ ./build update
$ ./build world</pre>
</p>
<p>
<b>Usage.</b> The <tt>bin</tt> directory contains scripts called <tt>java</tt>, <tt>javac</tt> (courtesy of ecj), <tt>javap</tt>, and <tt>javah</tt> that run Lisp with the right arguments.
</p>
<pre>$ ./bin/java -version
CLOAK Virtual Machine, running on SBCL 0.9.8.6 (Linux 2.6.22 X86)
Copyright (C) 2003-2007 David Lichteblau
</pre>
<p>
Technically it is a precompiler, and to avoid unpleasant surprises at run time, you might want to run
<pre>./bin/precompile foo.jar</pre>
before starting anything non-trivial.
</p>
<p>
Finally, read cloak/TODO and start hacking.
</p>
<p><b>What's new?</b> Compared to the big binary tarball available previously, this one comes with sources only, has been updated for current SBCL, and for Classpath 0.91 (which is still ancient, but a little step forward). The scripts in bin/ are also new.
<p>
<br>
<sup>1</sup> No AMD 64 support yet. For now, use an x86 chroot instead.
</p>David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com2tag:blogger.com,1999:blog-5568788882760066320.post-61642450468919875072007-08-05T14:34:00.001+02:002007-08-05T14:34:51.388+02:00A new data structure for XML<a href="http://lichteblau.blogspot.com/2007/07/theres-exactly-one-way-to-do-it.html">As mentioned earlier</a>, I set out to define an alternative to the W3C's Document Object Model, inspired heavily by Java's XOM, but made for Common Lisp.<br/>
<br/>
The result is STP, a data structure for XML that is full-featured and uses CLOS, but is more natural than DOM and gets namespaces right. Its implementation <a href="http://www.lichteblau.com/cxml-stp/">cxml-stp</a> is available as an add-on library for Closure XML.<br/>
<br/>
(For most purposes, it should be preferable to other alternatives, but DOM fans -- in case there are any -- can be assured that DOM support in cxml will not go away either.)<br/>
<br/>
Read more about STP in the <a href="http://www.lichteblau.com/cxml-stp/tutorial/">tutorial</a>.David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com2tag:blogger.com,1999:blog-5568788882760066320.post-53181698469028638862007-07-28T21:16:00.000+02:002007-07-28T21:26:00.810+02:00Macros for XSLTGary King is <a href="http://metabang.com/unclogit/?p=150">confused because [XSLT] seems so ridiculously verbose</a>. Others have already suggested <a href="http://www.idealliance.org/papers/extreme/proceedings/html/2006/Novatchev01/EML2006Novatchev01.html">mad</a> higher-order function tricks using XSLT 2.0.<br/>
<br/>
My solution: Macros. If XSLT lacks an element to do what you want (creating a text node with a newline, in Gary's case), just invent the feature you need and send your XSLT stylesheet through <i>another</i> XSLT stylesheet to implement it.<br/>
<br/>
<b>BR</b><br/>
<br/>
Say you have <a href="http://www.lichteblau.com/blog/br/demo.xsl" style="color: green">demo.xsl</a> which wants to use<br/>
<pre style="color: green"> <x:br/></pre></br>
to emit a newline. (In this example, `x' is simply the namespace for our extensions.) Write an additional stylesheet <a href="http://www.lichteblau.com/blog/br/macros.xsl" style="color: blue">macros.xsl</a> and send the original <tt style="color: green">demo.xsl</tt> through the macro stylesheet to generate the actual XSLT source code. A macro template for <x:br> would be as simple as:<br/>
<pre style="color: blue">
<xsl:template match="x:br">
<span style="color: black"><_xsl:text></span><xsl:text><span style="color: black">&#10;</span></xsl:text><span style="color: black"></_xsl:text></span>
</xsl:template>
</pre>
In the macro stylesheet, <tt style="color: blue">xsl</tt> is the namespace of the "macro definition" and <tt>_xsl</tt> is the namespace of the "macro expansion". (If you care about details, the trick is to use <tt style="color: blue">xsl:namespace-alias</tt> to make the XSLT processor believe they are different namespaces.)<br/>
<br/>
<b>DOTIMES</b><br/>
<br/>
For a more interesting example of macro use, suppose we want to repeat our code <em>count</em> times. Doing this kind of iteration involves a recursive template call, which we want to hide. We will define a macro <tt><x:dotimes></tt> that can be used like this:<br/>
<pre style="color: green">
<x:dotimes var="i" count="3">
<xsl:value-of select="$i"/>
</x:dotimes>
</pre>
Our macro stylesheet replaces each use of <tt><x:dotimes></tt> with a template call, and adds a recursive template as a top-level element:<br/>
<pre><span style="color: blue"> <xsl:template match="xsl:stylesheet">
<xsl:copy>
<xsl:apply-templates select="@*|node()"/>
<xsl:for-each select="//x:dotimes"></span>
<_xsl:template name="x:dotimes_{generate-id()}">
<i>... recursive template definition here ...</i>
</_xsl:template><span style="color: blue">
</xsl:for-each>
</xsl:copy>
</xsl:template>
<xsl:template match="x:dotimes"></span>
<_xsl:call-template name="x:dotimes_{generate-id()}">
<i>... parameters elided for brevity ...</i>
</_xsl:call-template><span style="color: blue">
</xsl:template></span></pre>
Download the full <a href="http://www.lichteblau.com/blog/br/macros.xsl">macros.xsl</a> and <a href="http://www.lichteblau.com/blog/br/demo.xsl">demo.xsl</a> to try the example. To run it with xsltproc, use the Makefile in the same directory.David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com2tag:blogger.com,1999:blog-5568788882760066320.post-46998262290222436282007-07-01T16:30:00.000+02:002007-10-26T22:05:14.944+02:00There's exactly one way to do it<a href="http://www.xom.nu/">XOM</a> is a DOM alternative written in Java and <i>for</i> Java -- in contrast to DOM, which feels wrong in almost every language.<br/>
<br/>
<a href="http://www.xom.nu/whatswrong/img0.html">Key phrases</a>:<br/>
<ul><li>"Comatose lists"</li> <li>This is a cathedral, not a bazaar</li> <li>There's exactly one way to do it</li><li>The Wrong Side of 80/20</li></ul>
<br/>
Lots of good ideas waiting to be stolen. Stay tuned for a Common Lisp adaptation.David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com4tag:blogger.com,1999:blog-5568788882760066320.post-60179477024875585682007-06-24T16:03:00.000+02:002007-07-28T20:04:34.539+02:00clbuild has a new homeAs Planet Lisp readers <a href="http://lukego.livejournal.com/2530.html">should</a> <a href="http://www.advogato.org/person/crhodes/diary.html?start=113">already</a> <a href="http://lukego.livejournal.com/3137.html">know</a>, Luke Gorrie's <i>clbuild</i> has been evolving slowly in at least three different darcs repositories for the last few months.<br/>
<br/>
Someone must have decided that it was time to give it a new home and created a shiny new clbuild project on common-lisp.net.<br/>
<br/>
This completely empty project looked a little sad and lonely though, so I gave it a <a href="http://common-lisp.net/project/clbuild">home page</a>.<br/>
<br/>
Hope you don't mind -- or if you do, you shouldn't have given me write access...David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com2tag:blogger.com,1999:blog-5568788882760066320.post-32383204118680198322007-05-27T01:25:00.000+02:002007-07-28T20:05:01.799+02:00Relax NG for cxml<a href="http://relaxng.org/spec-20011203.html">Relax NG</a> is the friendly schema validation standard for XML with clean namespace support, data types, and uniform treatment of attributes. It is closed under union (allows ambiguity). And it offers a compact non-XML syntax.<br/>
<br/>
Released today, <a href="http://www.lichteblau.com/cxml-rng/">cxml-rng</a> is my implementation of Relax NG in Common Lisp, as an extension to <a href="http://www.common-lisp.net/project/cxml">Closure XML</a>.<br/>
<br/>
Learn Relax NG through van der Vlist's <a href="http://books.xmlschemata.org/relaxng/page2.html">book</a>, read <a href="http://www.thaiopensource.com/relaxng/design.html">The Design of Relax NG</a> by James Clark, <a href="http://www.grappa.univ-lille3.fr/tata/">pump subtrees</a>, or try cxml-rng yourself.David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com0tag:blogger.com,1999:blog-5568788882760066320.post-15517438114023459752007-03-28T10:28:00.000+02:002007-07-28T20:05:29.734+02:00Federal trojan horse(Apologies for non-Lisp content.)<br/>
<br/>
German police is going to install surveillance software on suspects' computers.<br/>
<br/>
They are going to install it online through the Internet without anyone noticing, and of course they will do it without exploiting security vulnerabilities. Here is what the president of the Federal Criminal Police Office <a href="http://www.taz.de/pt/2007/03/26/a0119.1/text">has to say about it</a> (in german):<br/>
<br/>
<b>taz</b>: How will the "Online-Search" of a computer work technically then?<br/>
<br/>
<b>Ziercke</b>: Naturally I cannot discuss that publically.David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com2tag:blogger.com,1999:blog-5568788882760066320.post-76827788080893263672007-03-10T20:15:00.000+01:002007-07-28T20:07:01.090+02:00clbuild on cygwin<b>Step 1: SBCL</b><br/>
<br/>
Install <a href="http://sbcl.sourceforge.net/platform-table.html">SBCL</a> using its Windows .msi installer.<br/>
<br/>
<b>Step 2: Cygwin and darcs</b><br/>
<br/>
Since clbuild is a shell script, you need to install cygwin, even though SBCL itself does not depend on it.<br/>
<br/>
Get it from <a href="http://www.cygwin.com/setup.exe">cygwin.org</a>. Make sure to select all packages that clbuild uses to download software. You will need at least <b>cvs</b>, <b>subversion</b>, and <b>wget</b>. In addition, you might want to install <b>emacs</b> (for slime) and <b>X</b> (for CLIM with the CLX backend).<br/>
<br/>
[EDIT: Don't use emacs from cygwin, install the native Windows port of Emacs instead.--2007-06-24]<br/>
<br/>
Not included with cygwin is <b>darcs</b>, but it has a cygwin port, so download it manually from <a href="http://wiki.darcs.net/DarcsWiki/CategoryBinaries#head-c7910dd98302946c671cf63cb62712589b392074">darcs.net</a> and add it to your $PATH.<br/>
<br/>
<b>Step 3: clbuild</b><br/>
<br/>
Cygwin support is new in my clbuild tree, so until another clbuild hacker merges those changes, fetch it from:<br/>
<br/>
<pre style="background-color: #cccccc">$ darcs get http://www.lichteblau.com/blubba/clbuild
</pre><br/>
<br/>
<b>Step 4: Bleeding edge</b><br/>
<br/>
ASDF as included with SBCL 1.0.3 does not work with clbuild, so you need to replace it with a version including my patch for Windows shortcut support.<br/>
<br/>
Download <a href="http://www.lichteblau.com/blubba/shortcut/asdf.lisp">asdf.lisp</a> and <a href="http://www.lichteblau.com/blubba/shortcut/asdf.fasl">asdf.fasl</a> and copy them into the <tt>asdf/</tt> subdirectory of your SBCL installation, replacing the original versions. (<a href="http://www.lichteblau.com/blubba/shortcut/asdf.diff">diff</a>)<br/>
<br/>
[EDIT: the asdf.fasl linked there isn't up-to-date anymore, but asdf.lisp and asdf.diff are still there, including some fixes. Drop them into your source tree and recompile SBCL. --2007-06-24]<br/>
<br/>
<b>Run clbuild</b><br/>
<br/>
That's it. Now just run clbuild:<br/>
<br/>
<pre style="background-color: #cccccc">$ cd clbuild
clbuild$ chmod +x clbuild
clbuild$ ./clbuild build
</pre>
<br/>
To run CLIM applications using the CLX backend, start an X server first and set $DISPLAY accordingly. (It appears to be necessary to specify an IP address in $DISPLAY so that CLX does not attempt a unix domain socket connection.)<br/>
<br/>
<pre style="background-color: #cccccc">clbuild$ X&
clbuild$ export DISPLAY=127.0.0.1:0
clbuild$ ./clbuild listener
</pre><br/>
<br/>
<b>Optional: Gtkairo</b><br/>
<br/>
To try CLIM's gtkairo backend instead, download GTK+ from <a href="http://downloads.sourceforge.net/gimp-win/gtk%2B-2.10.6-1-setup.zip">gimp-win.sf.net</a>. (For some reason, the installer is wrapped in a zip file.)<br/>
<br/>
Add the <tt>bin</tt> directory of that GTK+ installation to your PATH and configure clbuild to use gtkairo:<br/>
<br/>
<pre style="background-color: #cccccc">clbuild$ export PATH="/cygdrive/c/Programme/gtk-2.10/bin:$PATH"
clbuild$ export CLIM_BACKEND=gtkairo
clbuild$ ./clbuild listener
</pre><br/>
<br/>
<a href="http://www.lichteblau.com/blubba/shortcut/listener-windows.png">ObScreenshot</a> of the listener.<br/>
<br/>
(Expect to find some gtkairo/Windows repainting bugs though.)David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com0tag:blogger.com,1999:blog-5568788882760066320.post-6484098943929055302007-03-03T12:34:00.000+01:002007-07-28T20:07:52.495+02:00Klacks parsing<a href="http://common-lisp.net/project/cxml">Closure XML</a> has been based on a SAX-like API for several years now (in addition to the DOM implementation on top of that). But although the pervasive use of SAX within CXML itself has been a success story, most users seem to prefer DOM usage over SAX handler hacks. Anyone who has ever parsed a non-trivial schema using SAX knows why: Maintaining separate <tt>start-element</tt> and <tt>end-element</tt> methods is very inconvenient. Code ends up dispatching on tag names using huge <tt>case</tt> forms while doing all bookkeeping manually in slots of the handler instance.<br/>
<br/>
Starting with the current release of CXML, there is now a new parser interface called <a href="http://common-lisp.net/project/cxml/klacks.html">Klacks</a>.<br/>
<br/>
Similar to StAX, the new interface is more convenient than SAX, while still providing the same features as the old one, including validation.<br/>
<br/>
Basically, the klacks parser can be used as a (rather sophisticated) tokenizer, and you get to write a recursive descent parser based on that.<br/>
<br/>
<b>SAX and StAX</b> are Java's protocols for XML parsing. Sometimes they are being referred to as low-level interfaces for "expert" use only (the suggested alternative being something like DOM), but their purpose is really to parse XML without building an in-memory representation.<br/>
<br/>
Low-level or not, they are the right choice when parsing into application-defined data structures or when performing simple on-the-fly transformation of XML data as it is being read.<br/>
<br/>
In SAX, an XML parser will process the entire document in one go, emitting events as it sees them. User code needs to implement its own handler class, with methods for the events it cares about. The SAX concept is known as "push-based".<br/>
<br/>
In contrast, the "pull-based" <a href="http://jcp.org/en/jsr/detail?id=173">StAX</a> parsing model is similar to working with an input stream. User code starts by creating an input stream object for the XML document, then reads events from that stream one by one. (Klacks uses the term <i>source</i> instead of <i>stream</i>, to avoid confusion with Common Lisp streams.)<br/>
<br/>
<b>API design choices.</b> StAX distinguishes between a high-level API, which creates a Java object for each event, and the low-level API, which just returns an enum indicating the type of event, and has separate methods to access the current event's data.<br/>
<br/>
Klacks has just one set of functions for both purposes, since it seemed more lispy to use multiple values. Instead of returning just a keyword indicating the event type, the main klacks functions always include useful event data as additional return values.<br/>
<br/>
Java's StAX also includes classes for XML serialization. No such extension was needed for CXML, since it already supports convenient serialization using SAX events. The <tt>with-element</tt> macro and related functions make generation of those events easy.<br/>
<br/>
<b>Simple klacks parsing example:</b><pre>* (defparameter *source* (cxml:make-source "<example>text</example>"))
* (klacks:peek-next *source*)
:START-DOCUMENT
* (klacks:peek-next *source*)
:START-ELEMENT
NIL ;namespace URI
"example" ;local name
"example" ;qualified name
* ...
</pre>David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com4tag:blogger.com,1999:blog-5568788882760066320.post-75562263310132478332007-02-24T12:30:00.000+01:002007-07-28T20:08:33.539+02:00McCLIM's tab layout<a href="http://www.paoloamoroso.it/log/">Paolo Amoroso</a> used to blog about McCLIM commits. But he stopped posting updates some months ago, so here is my own report, starting with the tab-layout.<br/>
<br/>
The original tab-layout, written by <a href="http://bl0rg.net/~mgr">Max-Gerd Retzlaff</a>, implements a CLIM pane similar to what GTK+ calls GtkNotebook, using only portable CLIM mechanisms to do so:<br/>
<br/>
<img border="0" src="http://www.lichteblau.com/img/generic-clim-tabs.png"><br/>
<br/>
With my changes, committed a few weeks ago, the tab-layout's architecture is now closer to other gadgets in CLIM, which are split into an abstract superclass and several frame-manager-specific subclasses. This new version allows the Gtkairo backend to implement its own subclass using a native GtkNotebook:<br/>
<br/>
<img border="0" src="http://www.lichteblau.com/img/gtkairo-clim-tabs.png"><br/>
<br/>
One aspect of the tab-layout that still stands out is the use of presentations and commands. While both are fundamental CLIM concepts, other gadgets defined in CLIM 2 come without any integration into the presentation system, so there was some discussion about turning the tab-layout into a "proper" gadget using simple callbacks instead of commands.<br/>
<br/>
In the end, I decided to keep the use of presentations in the generic tab layout and hack its Gtkairo version to simulate them, too. To me, the greatest advantage of this implementation is that, thanks to presentations, there is an easy way to define context menus for tab pages in a CLIMy way.<br/>
<br/>
You can try the new-and-improved tab-layout using simple demo code included with clim-examples (start it using <tt>(clim-demo::demodemo)</tt>), or by trying one of currently two real applications featuring tabs. One of them is <a href="http://common-lisp.net/project/beirc/">beirc</a>, the other is the web browser <a href="http://common-lisp.net/project/closure/">closure</a>, which now supports tabbed browsing!David Lichteblauhttp://www.blogger.com/profile/05088343424853670990noreply@blogger.com0