BA 372 - XML Lab

In this lab we'll explore several ways of consuming and producing XML, both as files and as Web services. The examples are based on material collected from a variety of places.

As discussed in class, the combined rigidity and simplicity of XML syntax make it relatively easy to parse; hence, a growing variety of parsing methods and models is available.

Study the following (DTDless) XML file and store it in c:\temp\family.xml:

<?xml version="1.0" encoding="UTF-8"?>
<family>
<person gender="male">
    <firstname>Don</firstname>
    <lastname>Hurst</lastname>
</person>
<person gender="female">
    <firstname>Mary</firstname>
    <lastname>Hurst</lastname>
</person>
</family>

Notice that whereas the firstname and lastname of a person are represented as atomic XML elements, the person's gender is stored as an attribute. As far as we know, there are no good rules for deciding which of these methods to use.

One of the classes in .Net for reading and parsing XML is the XMLReader (example below based on XML File Parsing in VB.Net by Pratik Desai):

IN VB.Net, start a new console application containing the following code:

Imports System.IO
Imports System.Xml

Class foo
Shared Sub Main()
    Dim m_xmlr As XmlTextReader

    'Create the XML Reader
    m_xmlr = New XmlTextReader("C:\temp\family.xml")

    'Disable whitespace so that you don't have to read over whitespaces
    m_xmlr.WhiteSpaceHandling = WhiteSpaceHandling.NONE

    'read the xml declaration and advance to the <family> tag
    m_xmlr.Read()

    'read the <family> tag
    m_xmlr.Read()

    'Read each of the persons
    While Not m_xmlr.EOF
      'Go to the <person> tag
      m_xmlr.Read()

      'if not start element exit while loop
      If Not m_xmlr.IsStartElement() Then
        Exit While
      End If

      'Get the 'gender' attribute value
      Dim genderAttribute As String = m_xmlr.GetAttribute("gender")

      'Read elements <firstname> and <lastname>
      m_xmlr.Read()

      'Get the <firstName> element value
      Dim firstNameValue As String = m_xmlr.ReadElementString("firstname")

      'Get the <lastName> element value
      Dim lastNameValue As String = m_xmlr.ReadElementString("lastname")

      'Write results to the console
      Console.WriteLine("Gender: " & genderAttribute)
      Console.WriteLine(" FirstName: " & firstNameValue)
      Console.WriteLine(" LastName: " & lastNameValue)
      Console.WriteLine()
    End While

    'close the reader
    m_xmlr.Close()

End Sub
End Class

Compile & run:

Study the code, especially the way the parsing is done. Notice that this way of programming the parsing of an XML document is awkward at best. The programmer must carefully specify each read, !!!in the proper order; i.e. sequentially!!! and according to the structure of the XML document. Things quickly get confusing and complicated.

For instance, assume that we'd have to process a slightly more complex data structure:

<?xml version="1.0" encoding="UTF-8"?>
<family>
  <person gender="male">
     <name>
        <firstname>Don</firstname>
        <lastname>Hurst</lastname>
     </name>
     <job>retired</job>
  </person>
  <person gender="female">
     <name>
        <firstname>Mary</firstname>
        <lastname>Hurst</lastname>
     </name>
     <job>dentist</job>
  </person>
</family>

Try modifying the above VB.Net code to read and parse this more complex data structure. (Try doing this before using the code listed below).

Imports System.IO
Imports System.Xml

Class foo
    Shared Sub Main()
        Dim m_xmlr As XmlTextReader

        'Create the XML Reader
        m_xmlr = New XmlTextReader("C:\temp\family.xml")

        'Disable whitespace so that you don't have to read over whitespaces
        m_xmlr.WhitespaceHandling = WhitespaceHandling.None

        'read the xml declaration and advance to <family> tag
        m_xmlr.Read()

        'read the <family> tag
        m_xmlr.Read()

        'Loop, reading the information on 'persons' in the family
        While Not m_xmlr.EOF

            'Read to the <person> tag
            m_xmlr.Read()

            'if not start element exit while loop
            If Not m_xmlr.IsStartElement() Then
                Exit While
            End If

            'Get the 'gender' attribute value
            Dim genderAttribute As String = m_xmlr.GetAttribute("gender")

            'Read the <name> tag
            m_xmlr.Read()

            'read whatever's inside the <name> tag (firstname & lastname elements>
            m_xmlr.Read()

            'Get the <firstName> element value
            Dim firstNameValue As String = m_xmlr.ReadElementString("firstname")

            'Get the <lastName> element value
            Dim lastNameValue As String = m_xmlr.ReadElementString("lastname")

            'read the <job> tag
            m_xmlr.Read()
            'Get the <job> information out.
            Dim jobValue As String = m_xmlr.ReadElementString("job")

            'Write results to the console
            Console.WriteLine("Gender: " & genderAttribute)
            Console.WriteLine(" FirstName: " & firstNameValue)
            Console.WriteLine(" LastName: " & lastNameValue)
            Console.WriteLine(" job: " & jobValue)
            Console.WriteLine()
        End While

        'close the reader
        m_xmlr.Close()

    End Sub
End Class

A better, easier to use class is the XmlDocument: (example below based on XML File Parsing in VB.Net by Pratik Desai):

The XmlDocument class is based on the Document Object Model (DOM) discussed in class. In this approach, the entire XML document is read into memory as a DOM object. The programmer can then randomly(!!) query any of the object's elements (note the similarity between this type of information accessing and RAM):

Imports System.IO
Imports System.Xml

Class foo
    Shared Sub Main()
            Dim m_xmld As XmlDocument
            Dim person_nodeList, name_nodeList As XmlNodeList
            Dim person_node, name_node As XmlNode

            'Create the XML Document
            m_xmld = New XmlDocument()

            'Load the Xml file
            m_xmld.Load("C:\temp\family.xml")

            'Get the list of name nodes
            name_nodeList = m_xmld.SelectNodes("/family/person/name")

            'Loop through the nodes
            For Each name_node In name_nodeList

                'Get the <firstName> element value
                Dim firstNameValue As String = name_node.ChildNodes.Item(0).InnerText

                'Get the <lastName> element value
                Dim lastNameValue As String = name_node.ChildNodes.Item(1).InnerText

                'Write result to the console
                Console.Write(" FirstName: " & firstNameValue)
                Console.Write(" LastName: " & lastNameValue)
                Console.WriteLine()
            Next

            'Get the list of person nodes
            person_nodeList = m_xmld.SelectNodes("/family/person")

            'Loop through the nodes
            For Each person_node In person_nodeList

                'Get the gender attribute value
                Dim genderAttribute As String = person_node.Attributes.GetNamedItem("gender").Value

                'Write result to the console
                Console.Write("Gender: " & genderAttribute)
                Console.WriteLine()
            Next

    End Sub
End Class

Notice in the code above how, when using the XmlDocument class, the whole XML document gets read into the m_xmld object.

Notice how, once stored in an XmlDocument object, we can randomly address (pull out) any of the branches of the XML tree (remember from class that all well-formed XML documents can be represented as trees).

In the above example we pull out the <name> branches before we pull out the <person> branch, even though <name>s appear inside <person>s.

Notice how we use a directory/folder-like syntax for addressing branches in the XML tree; e.g., /family/person/name or /family/person.

This convention for referring to nodes and elements in a DOM tree is defined in XPath (XML Path Language) which is part of the XML standard.

Before moving on to the next exercise, try modifying the above code to 'pull out' some other components. Make sure you're comfortable with the Xpath syntax before moving on.

Now we have an idea of how to read and parse XML files, let's read and parse a 'real' one from the Web; i.e., one from www.teachengineering.com.

Point your browser to www.teachengineering.com.

Click on the Search Curriculum menu option on the left.

Fill out the search form with the search string spaghetti.

Click on the first lesson listed under the lessons category (Boxed In and Wrapped Up).

Notice the URL associated with this lesson:

http://www.teachengineering.com/view_lesson.php?url=http://www.teachengineering.com/collection/duk_/lessons/duk_boxes_mary_less/duk_boxes_mary_less.xml

The URL references www.teachengineering.com twice:

First as part of the document address: http://www.teachengineering.com/view_lesson.php

Second as the value of the url variable passed to view_lesson.php using the GET method.

Notice that the url variable points to an XML document:
http://www.teachengineering.com/collection/duk_/lessons/duk_boxes_mary_less/duk_boxes_mary_less.xml

Point your browser to this XML document and confirm that it is, in fact an XML document.

Notice that it specifies elements such as a <summary> and <time>.

Now let's write a program that

pulls the above XML file from the TeachEngineering server, and

pulls out its <summary> and prints it to the screen.

Try:

!!!Note that this does not work!!!, even though we use the exact same code model as used previously.
The problem here rests in the so-called namespace declared in the XML document (you might have to pull up the page source to see this):

<lesson xmlns="http://www.teachengineering.org"...

A word on XML namespaces:

Since XML is an extensible language and anyone can arbitrarily define a vocabulary of an XML specification, it is quite likely that the same element or term appears in different XML specifications, each with a very different meaning. For instance, notice that our TeachEngineering document has a <title> tag. However, many XML documents on the Web not related to TeachEngineering also have a <title> tag, e.g., most (X)HTML documents have such a tag. Yet the meaning of these tags is quite different.

From this, we can infer that we need a way to distinguish the <title> tag in a TeachEngineering document from <title> tags in nonTeachEngineering documents. XML namespaces do just that.

An XML namespace is a string that we can use to uniquely qualify any of the XML tags in our documents. For instance, all TeachEngineering documents could declare this unique namespace string as follows:
xmlns:te="http://www.teachengineering.org"
The above line declares that when used in a tag in the document, the so-called namespace prefix te must be replaced with the full namespace http://www.teachengineering.org.

e.g., the entry <te:title>, would really stand for <http://www.teachengineering.org:title>

To now use this namespace to make each and every tag in the document have a truly unique name, we can use the syntax <namespace prefix:tagname>; e.g., <te:title>.

Now, when you look at our code above, you'll notice that although in the TeachEngineering XML file we specify the document's namespace, we do not incorporate any namespace information in the Xpath expressions. For instance, the statement

my_node = m_xmld.SelectSingleNode("/lesson/summary", names)

does not specify a namespace, and hence, the parser cannot match the above Xpath expression with the document.

To make matters worse, the xmlns="http://www.teachengineering.org" namespace does not specify a namespace prefix such as in
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

To correct for this problem, we'll give the http://www.teachengineering.org name space a "dummy" prefix and include that in the Xpath expression:

Imports System.IO
Imports System.Xml

Class foo
    Shared Sub Main()

        Dim m_xmld As XmlDocument
        Dim my_node As XmlNode

        'Create the XML Document
        m_xmld = New XmlDocument()

        'Load the XML file
        m_xmld.Load("http://www.teachengineering.com/collection/duk_/lessons/duk_boxes_mary_less/duk_boxes_mary_less.xml")

        'Get me a namespace manager object
        Dim names As XmlNamespaceManager = New XmlNamespaceManager(m_xmld.NameTable)
        'Set a "dummy" namespace prefix
        names.AddNamespace("dummy", "http://www.teachengineering.org")

        'Get the <summary> node including the 'dummy' namespace prefix. Notice how the 'dummy' is used on each of the             'branches in the DOM tree.
        my_node = m_xmld.SelectSingleNode("/dummy:lesson/dummy:summary", names)

        Console.WriteLine(" Summary: " & my_node.InnerText)
        Console.WriteLine()

    End Sub
End Class

A few things to notice:

Notice how whereas in previous examples we passed a regular file path into Load(), we now pass in a URL; no need to first do an HTTP request and then read the XML from the body of the HTTPResponse. The Load()routine is smart enough to recognize that the XML file must be retrieved through HTTP.

Notice how we 'add' a dummy namespace prefix to the http://www.teachengineering.org namespace.

Notice how we use the prefix in the Xpath expression:

my_node = m_xmld.SelectSingleNode("/dummy:lesson/dummy:summary", names)

Serialization: XML-OO transforms:

Given the hierarchical (nested) structure of both XML and object orientation (recall that in OO classes can themselves contain classes), would it not be nice if we could transfer easily between the world of XML (tags nested within other tags) and the world of object orientation. It would, for instance, be nice if we could read the above TeachEngineering lesson directly from XML into a object of class 'Lesson' stored in memory or vise versa.

Serialization: object --> XML.

Deserialization: XML --> object.

To demonstrate these mechanisms, we'll use a simpler object than a TeachEngineering lesson. Take a look at the following (serialization) code:

Using standard OOP codes, we define a person as a class with three components:

a cName object of class Name, which itself consists of two components:

first_name.

last_name.

a cAge.

a cProfession.

In Main(), we declare and initialize customer as a person object.

Next, we create an XmlSerializer object and pass it the information about(!!) a person (this allows the serializer to figure out the components of the person class). Note that we do not(!!) pas it the object customer. Instead, we pass it the class(!!) person. After all, it's the class definition that holds the information on who or what a customer really is.

Finally, we ask the serializer to serialize our instantiated person (customer) into XML and we feed the result to the screen.

Compile and run.

Notice that the result is the corresponding XML fragment written to the screen.

Now let's go the opposite direction (deserialization):

Store the following XML file in c:\temp\customer.xml

<?xml version="1.0" encoding="utf-16"?>
<Person xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<cName>
<first_name>Mary</first_name>
<last_name>Hurst</last_name>
</cName>
<cAge>50</cAge>
<cProfession>dentist</cProfession>
</Person>

Now let's read this XML file and deserialize it into a person object.

Imports System.IO
Imports System.Xml
Imports System.Xml.Serialization

Public Class Name
    Public first_name As String
    Public last_name As String
End Class

Public Class Person
    Public cName As Name
    Public cAge As Integer
    Public cProfession As String
End Class

Class foo
    Shared Sub Main()

        Dim serialization As String
        Dim string_reader As StringReader

        Dim xml_serializer As New XmlSerializer(GetType(Person))

        Dim Reader As IO.StreamReader
        Dim InFile As IO.FileStream = Nothing

        'Open the stream by hooking it up to the file:
        Try
            InFile = New IO.FileStream("C:\temp\customer.xml", IO.FileMode.Open, IO.FileAccess.Read)
        Catch e As Exception
            Console.WriteLine(e.Message)
            Console.WriteLine("Problems opening file...")
            System.Environment.Exit(1)
        End Try

        'Connect the StreamReader to the stream:
        Reader = New IO.StreamReader(InFile)

        'Read in the string
        serialization = Reader.ReadToEnd()
        InFile.Close()

        ' Make a StringReader holding the serialization.
        string_reader = New StringReader(serialization)

        ' Create the new Person object from the serialization.
        Dim customer As New Person
        customer = CType(xml_serializer.Deserialize(string_reader), Person)

        'Write the contents of the customer to the screen
        Console.WriteLine("first name: " + customer.cName.first_name)
        Console.WriteLine("last name: " + customer.cName.last_name)
        Console.WriteLine("age: " + CStr(customer.cAge))
        Console.WriteLine("profession: " + customer.cProfession)
        Console.WriteLine()
    End Sub
End Class

Notice how we read the entire contents of the XML file into a string called serialization.

Notice how with the line

customer = CType(xml_serializer.Deserialize(string_reader), Person)

we deserialize the XML string into a person object called customer.

When to you use this (de)serialization technique?

When you want to expose internal data as an XML Web service.

When you want to consume an XML Web service into an OO program.