I've been working on a Pet Project for some time now. It's written in Java, using NetBeans, Swing and with Scripting done in Groovy. It's a very generic Transaction Reader and Simulator.
Why? It is just Fun! I'm learning a lot by working on it, and its pretty useful, at least for me.
A Transaction is either a specific kind of a message, or a record in a file.
These transactions are Financial transactions (mostly). But the project is generic enough to handle any message or record, as long as you know what the messages are supposed to be like.
Messages can be any kind of format. I am currently working on BASE24 ISO, BASE24 ATM and POS Internal messages, and some ATM native messages. Other messages like VISA, MasterCard, or any Switch can be easily added to the system. More on this later.
Currently, the system can read ACI XPNET Audit Files, and any Tandem Enscribe file. Just transfer the files to the PC in Binary (or use the built-in transparent FTP feature in the program) to read all records and expand them.
So, what can you do with these messages? Well, the main interface displays the expanded messages as a cool looking Tree, XML, or Hex Dump. You can search the files for any text, or any value in any field. You can output the data as nice XML, or CSV. Or even write a script to put the data to an RDBMS. I used it to extract BASE24 TLF fields (including some Tokens) to an RDBMS for analysis.
The Simulator is still in its early stages under development, but so far, it can open TCP sockets and talk back to a BASE24 host in BASE24 ISO-8583 messages. The Simulator is actually a Groovy script that talks in high level using Structures. It does NOT do any of the cumbersome ISO message packing at all. That is all done by the Message Templates. All you need the Groovy Script is something like:
MSG.dump()
if(MSG.MSG_TYP.value == "0800") {
MSG.MSG_TYP.value = "0810";
bits = MSG.BIT_FIELDS;
// Add the response code field
resp = MSG.BIT_FIELDS.add("39.RESP_CDE");
// and set to 00
resp.value = "00";
// then reply
SRC.write(MSG)
// and we are done.
return true;
}
One thing that I'm planning to do is have the program read some file (Audit or Transaction Log) and create messages to send to a receiver. That will be fun! and will be very useful for testing and trouble-shooting.
Literally anything is possible. Really! I used that!
Okay so how do you define a Message? All messages are defined in XML, and human readable XML.
The XML describes what the message looks like. The program will do the conversion in and out. You do not write any line of code to expand any message. A sample XML snippet that describes a message is:
That XML should be readable and maintainable by humans.
1 <Struct name="B7">
2 <BNumber size="8" name="RBA"/>
3 <FString size="35" name="TLF_NAM"/>
4 <FString size="1" name="TKN_RETRV_OPT"/>
5 <Struct name="ATM_KEY">
6 <Struct name="CRD">
7 <FString size="4" name="LN"/>
8 <FString size="4" name="FIID"/>
9 <FString size="19" name="PAN"/>
10 <FString size="3" name="MBR_NUM"/>
11 </Struct>
12 </Struct>
13 </Struct>
I created some utilities in the program itself to convert DDL (the DDLTAL kind) to XML, because Tandem does not give me XML of its data structures. And because I am too lazy to convert it by hand. I'd spend 2 weeks automating something I can manually do in 2 hours. That's just me.
Which brings me to Xerialize. To create and read the XML, I initially worked with JAXB. I also tried XStream and Simple. Those did not give me the flexibility I needed, specially with the constructs I have. JAXB was not pleasant, but it the job. It is very picky about certain things, specially about polymorphic lists, which i heavily rely on. When I sub-classed some base class, some properties where written to XML twice!. But the last straw was when I upgraded to JDK 1.6u5 (I used u3). My XML was not readable at all. Something was broken, and not just for me. I posted an issue on the JAXB list, and some other people had the same issue. And the issue is not even looked at for a month now!
Screw JAXB, I'll create my own XML. And so Xerialize was born. About four days ago. Xerializer is a very generic library for serializing Java objects using Annotations.
You just annotate the getters and setters they way you need them, and XML is written, and read back. Your annotated getters and setters are called as usual when reading and writing. Discover is all done at runtime using Reflection. Anything that is not annotated is skipped.
No XJC, no Schema compiler, no nested XmlElment for XmlElements. Plain and simple. I currently use DOM to do the XML work, but may change that later.
These are the general rules for using Xerialize:
- The tag name is the Java class name, except for some nested lists You can qualify the class name's package using XML namespace. So in the above sample, FString is a Java class.
- Simple built-in class properties should be XML attributes. Annotate the ones you need.
- Object properties are XML Elements. Annotate the ones you need.
- Generic or Polymorphic lists should be handled without any extra configuration. If the FStringV is a sub-class of FString, then I can put that XML element and the reading writing just works.
- I know what methods to use for getters and setters, so Xerialize will only use what I annotate. You can actually hide an annotated property in a subclass by overriding the getter and setter and not annotate them.
- Default values are null during writing XML. Any null element will not be written. If an element is not in the XML, then the setter will not be called.
- Must have public default constructor.
- All annotated methods must be public. Xerialize does not use magic to look at private fields or methods.
- XML must support namespaces and XInlcude.
- Xerialize must support an easy to use method for nested Tree like nodes. A node containing a list of other nodes and so on. All these nodes are of one subclass. In the above example, Struct is superclass for both BNumber and FString.
9 comments:
Have you looked at STAX instead of DOM? It's much faster than DOM, and not nearly as memory hungry, but simpler than SAX.
I noticed that you had tried JAXB, XStream and Simple. XStream is a serialization library and I can't speak for Simple. This is a pretty well treaded problem space, and there are other extremely mature JAXB alternatives. Castor is one that immediately springs to mind, but there are others like JiXB and XMLBeans. Alternatively, you could always contribute the patch to the JAXB implementation yourself.
Can you tell me what you didn't like about Xstream. I have found it to be pretty reliable...
I've been tempted to do my own in the past as well. But that was before I knew of Castor and XStream.
Why not use Simple? How do your constructs not map?
It always amazes me that people would rather roll their own than contribute. We don't need another xml serialization library.
Thanks for all the comments.
I tried Simple and XStream first before going for JAXB. Since this is a Pet Project, I only have that much time to work on it, and there are usually big gaps.
Anyway. as far as I remember, XStream seamed more like a standard Java Serializer, that would map out all fields (not what I really wanted). Simple did not have that much flexibility as well. Maybe I did not spend that much time on either.
My Objects are slightly complex, with many getters and setters that should NOT be serialized. The other thing that is needed in my case is ease of use for polymorphic lists and subnodes.
THe best thing about writing a new library is learning something new. I learned a lot about reflection, and some cool XML stuff like namespaces and XIncludes.
I never looked at Castor, I may do that later. I'll also have a look at StAX. Standard Java libraries are preferred in my case to avoid getting much third party libraries and dealing with any license issues.
I'll try and clean up my code a bit to publish the library and post a simple how to.
Is this open source or available in any way. I find it hard to believe that Simple was not flexible enough given that you use annotations also, all elements and attributes can be intercepted and modified at runtime using a strategy, and objects can receive callbacks during serialization and deserialization to prepare themselves.
Also, I know from using JAXB and XStream that these can not handle very complex cycles, particularly involving arrays.
It would be good to see what problems you overcame that the frameworks you mention did not.
One very important missing feature that I required, and all except JAXB missed, was support for Xinclude. Xinclude is needed because many of the sub-structures I need are duplicated in others, and replicating the each sub-struct in many structures is not a good idea. I could be wrong, though.
As for an open release, I will hopefully work on releasing the latest sometime.
JAXB was used at initial stages until I upgraded to JDK 6u6, and then unmarshalling just did not work, after I created a few dozen XML documents.
I also did some tests with SAX and StAX, and for the size of my documents (1k - 20k max), DOM was almost as fast, especially that I needed most of the document data, and the created objects had more data than the XML documents.
You may also want to look at vtd-xml, the latest and most advanced XML processing
API
http://vtd-xml.sf.net
Post a Comment