 | Level: Intermediate Brett McLaughlin (brett@oreilly.com), Author, O'Reilly and Associates
02 Apr 2003 This tip begins to detail ways to handle large XML documents. You will learn what an XMLFilter is, and how it builds upon the core SAX API to offer advanced data filtering. This is the first piece in the puzzle of handling large datasets, allowing you to extract only relevant data from an XML document for output.
In the last tip, I examined several popular options for handling and ultimately outputting large XML datasets. It quickly became apparent that SAX, DOM, JAXP, JDOM, dom4j, and alternative XML APIs (particularly in-memory ones) were all poorly suited to dealing with giant XML datasets -- ones where records numbered in the tens of thousands. The only option that did work was to output characters to raw I/O streams -- and this solution had its own set of problems: It doesn't accurately represent XML, it's error-prone, and you can't navigate the XML tree.
In this tip, I begin the process of laying out a better option, albeit one that is little-understood and rarely used. However, like teaching any new API, it's going to take a little bit of time -- I'll lay out the basics in this installment, and then show you some practical applications of filters. Then you'll look at the basics of XMLWriters, and finally apply that API as well. When finished with these first five tips, you'll be equipped to filter out irrelevant data, and write out the data that is left to you, all within an XML-based framework.
The XMLFilter class
The class that I focus on in this tip is part of the SAX distribution, although it's generally overlooked or
flat-out ignored by most developers. Get a distribution of SAX, and look at the included classes; the one you want is org.xml.sax.XMLFilter. You should also ensure that you have the SAX helper classes, found in the org.xml.sax.helpers package. If you don't, you should be able to get these from the SAX project online (see Resources). In that package, you will want to focus on the org.xml.sax.helpers.XMLFilterImpl class. The XMLFilter interface and the XMLFilterImpl implementation of that interface pair add up to powerful filtering for your SAX-based applications.
If you crack open the source for XMLFilter, you'll find that it extends the
org.xml.sax.XMLReader interface, and adds two new methods:
-
public void setParent(XMLReader parent);
-
public XMLReader getParent();
This probably doesn't look like much; of course, you also get all the other XMLReader methods like startElement(), endPrefixMapping(), and so on. In each of these methods, you can operate upon the input XML data before an application gets to it.
To put this in perspective, remember that application code doesn't start to work on the XML data from a SAX parse until after that parsing has completed. However, you can insert an XMLFilter into the processing chain before this completion, meaning you get to modify data before the application gets that data (and, for example, outputs it). Since you have all of the SAX callback methods that an
XMLReader does, you can work with the elements, the attributes, the prefix mappings, and anything else that SAX can work with.
A simple filter
Let's look at a concrete example of an XMLFilter, so you can start to get an idea of what I'm talking about. Listing 1 shows a very simple SAX filter that removes all of the elements and attributes with a given URI from the input document. This effectively strips out all XML constructs within the given namespace.
Listing 1. A simple XMLFilter
import org.xml.sax.Attributes;
import org.xml.sax.SAXException;
import org.xml.sax.XMLReader;
import org.xml.sax.helpers.XMLFilterImpl;
public class RemoveNamespaceFilter extends XMLFilterImpl {
/** The URI of the namespace to remove */
private String namespaceURI;
public RemoveNamespaceFilter(XMLReader reader, String namespaceURI) {
super(reader);
this.namespaceURI = namespaceURI;
}
public void startElement(String uri, String localName, String qName,
Attributes atts)
throws SAXException {
// If this element is in the namespace to remove, skip it
if (namespaceURI.equals(uri)) {
// Do nothing, which skips this element
} else {
// Make a copy that we can modify
org.xml.sax.helpers.AttributesImpl newAtts =
new org.xml.sax.helpers.AttributesImpl();
// Remove (don't add to the new list) any attribute with the given name
for (int i=0; i<atts.getLength(); i++) {
String attURI = atts.getURI(i);
if (attURI.equals(namespaceURI)) { // skip it
} else {
newAtts.addAttribute(attURI, atts.getLocalName(i),
atts.getQName(i), atts.getType(i), atts.getValue(i));
}
}
// Delegate to the normal parsing behavior
parent.startElement(uri, localName, qName, newAtts);
}
}
public void endElement(String uri, String localName, String qName)
throws SAXException {
if (namespaceURI.equals(uri)) {
// Do nothing, which skips this element
} else {
// Delegate to normal parsing behavior
parent.endElement(uri, localName, qName);
}
}
|
First, notice that this class extends XMLFilterImpl instead of directly implementing XMLFilter. XMLFilterImpl provides a default version of all the required methods, allowing you to implement (override) only those methods that have customized behavior. This keeps your code cleaner, and requires less work on your part.
The key thing to understand here is how data is affected. If you keep in mind that this filter gets the input data before the registered XMLReader, it's all a piece of cake. This means that if you want the data to be handled by the registered reader, you simply delegate to it. For example, your startElement() method would simply call its parent's version of the same element, much like a constructor calls super(). In fact, all of the default methods in XMLFilterImpl() do just that. Listing 2 shows a couple of methods just so you get the idea.
Listing 2. Sample methods from XMLFilterImpl
public void startElement(String uri, String localName, String qName,
Attributes atts) throws SAXException {
parent.startElement(uri, localName, qName, atts);
}
public void endElement(String uri, String localName, String qName)
throws SAXException {
parent.endElement(uri, localName, qName);
}
|
This just passes on the data, unchanged, to the reader. However, as you saw in Listing 1, you can insert your own logic in these methods, overriding the default versions. To avoid the data being seen by the reader, simply avoid delegating to the reader's methods. In other words, a method with no body effectively ignores all content that the method deals with. However, be careful in these situations -- it is easy to pollute the data being sent back to the reader. For example, consider the case where you don't delegate in the startElement() method for certain data, but forget to do the same in endElement(). The result would be that some elements would never be reported as starting, but would be reported to the reader as ending. This would cause, in the best case, program errors, and in the worst case, data loss or corruption in your application.
Setting up a processing chain
Once you have your filter set up and compiled, you need to create a chain of processing. This should move from input document to filter to reader. You may even have multiple filters, stacked upon each other. As long as input comes first, and your reader (with application-specific callbacks) comes last, things work fine. However, you may have a particular order for your filters, and you should pay attention to that closely. Listing 3 shows how to set up your program for using filters.
Listing 3. Setting the processing chain
XMLReader reader = XMLReaderFactory.createXMLReader(vendorParserClass);
RemoveNamespaceFilter filter = new
RemoveNamespaceFilter(reader, "http://www.ibm.com/developer");
ContentHandler contentHandler = new ApplicationContentHandler(someDataParameter);
ErrorHandler errorHandler = new
ApplicationErrorHandler(someOutputStream, someErrorThreshold);
filter.setContentHandler(contentHandler);
filter.setErrorHandler(errorHandler);
InputSource inputSource = new InputSource(myXmlUri);
filter.parse(inputSource);
|
Notice that because the one or more filters must sit between input and the reader, all the operations that you would normally invoke on the reader are invoked on the filter. It then delegates any data that passes through the filter to the reader, as you saw in Listing 1. Obviously, you'll need to use your own variables in Listing 3, including content handlers, error handlers, entity resolvers, lexical handlers, and so forth -- however, I've left in dummy names to give you an idea how things fit together.
Between Listing 1, Listing 3, and this discussion, you should have enough information to write your own filters. I encourage you to do so, as SAX is already a bit of a tricky API, and filters sometimes add to the confusion. Get familiar with filters before moving on to the more advanced material in the upcoming tips.
Theory to practice
So far, I've dealt with XMLFilters largely as a matter of theory. You've seen what filters are, how the relevant SAX classes work, and how to attach filters to input chains when reading and parsing XML. However, this probably doesn't help you in cutting down on the data you have to output. The next tip will address that topic specifically, showing some example filters that will pull out attributes in a document, or pull out elements, remove namespaces, and a lot more. So hang in there for a few days, get comfortable with filters, and you'll put them to work in the next tip. Until then, I'll see you online.
Resources
About the author | 
| |
Brett McLaughlin has been working in computers since the Logo days (Remember the little triangle?). He currently specializes in building application infrastructure using Java-related technologies. He has spent the last several years implementing these infrastructures at Nextel Communications and Allegiance Telecom, Inc. Brett is one of the co-founders of the Java Apache project Turbine, which builds a reusable component architecture for Web application development using Java servlets. He is also a contributor of the EJBoss project, an open source EJB application server, and Cocoon, an open source XML Web-publishing engine. |
Rate this page
| |
