WSE: The Kerberos Token Cache
At the risk of being accused of WSE-bashing, here's one more post. It's interesting to see what you find out when you profile a windows service over time and see the footprint gradually increase.
It turns out that the KerberosTokenManager maintains a token cache of 2048 tokens by default. I can only assume that the intent here is to prevent replay attacks. Interestingly, though, replay detection is off by default. Now, a Kerberos Token is not exactly a lightweight object. Ours weigh in at 2,275 bytes including child references. That's over 4 MB of token cache.
Now, to be fair, Kerberos Tokens do expire, and the cache does perform a flush of expired tokens whenever a new one is added. But the default lifetime of a Kerberos Ticket is 10 hours. Even a moderately busy service can easily reach the 2048 token maximum size before even a single expiration occurs if the default lifetime is used.
"OK," you say. "What's 4 MB these days anyway?" That's fair -- it's not that much. But the real kicker is what happens when the cache limit is reached and a new token is added. WSE performs a flush, and if the size post-flush is still at or above the max size, WSE just sends an entry to the application event log and adds the token to the cache anyway!
So, if your WSE-based service is handling a high volume of messages and you're using the default Kerberos implementation with default ticket lifetime, this can behave like a memory leak until the first token expirations. The cache size will increase unchecked until the first tokens begin to expire. Let's say you receive one message every two seconds - 30 messages / minute. That's 18,000 messages before the first message expires. At 2,275 bytes / token, the cache has a footprint of 40 MB.
This is all well and good, but (1) it isn't documented anywhere, and (2) a cache is being built that is never used.
WSE Tracing Filters: Just Say No
Let me start by saying that Microsoft's WSE (Web Service Extensions) is a great framework. We've using its messaging API to build a foundation for a service architecture that should transfer relatively easily to Indigo, and its functionality is, on the whole, impressive.
That said. I have to wonder what the team was thinking in some areas. For example, the tracing filters are pretty much unusable except for very brief "let me take a peek" tracing sessions. We discovered this as our QA group was doing functional testing of our SOAP-based windows services. We delivered builds to QA with tracing turned on, and also included Simon Guest's WSE Trace Tool in the build so they could troubleshoot any message delivery problems they might run into. What a bad idea that was!
It turns out that in order to dump fully-formed XML documents to disk as the input and output traces, the filters just append each incoming or outgoing message to a private member XML document. Each filter also maintains a private filestream member. Each time a message enters or leaves the WSE pipeline, the stream pointer is set to 0 and the entire XML document is saved to disk. If you don't believe me, just use Reflector and take a look for yourself.
It doesn't take higher math to figure out that this very quickly results in a huge memory footprint and a thrashing GC. The size of the document very quickly makes its strings candidates for the large object heap, and every disposed string is larger than the previous one. With no compaction in the LOH, this creates a very interesting situation that is the GC equivalent of trying to put square pegs into round holes. After a very short time with tracing turned on, one of our test machines was maintaining two XML documents of over 100 MB each, and SciTech's .NET Memory Profiler (awesome tool) showed a GC that was flopping around like a fish on the bank. I'd sure like to hear some rationale for such a silly design decision.
About the Title
Almost a year ago now, I read a great article in Fast Company magazine. It was about design and meaning -- something I've been interested in ever since one of my college art professors handed me a copy of Ben Shahn's The Shape of Content. You can read the article here, but a key idea -- and I can't tell you how relieved I was to see that the business world is finally getting it -- is that design is meaning. Design permeates everything.
Strangely, in the world of software development, there is a rebellion against this idea, and I'm reasonably sure it stems from its computer science heritage. For some reason, science has always positioned itself as being somehow contrary to the world of aesthetics. Scientists don't design, they analyze. But analysis, if it has any worth beyond the simple establishment of a point of knowledge, is always part of a larger process of design. We analyze in order to gather knowledge that enables us to gain understanding. We are always seeking either to understand a design or to create one.
Many programmers reject the notion that they are designers, but the truth of the matter is that they design whether they acknowledge it or not. They may have designed software as they were coding it, but in the end the design is there in the product whether or not its creation was intentional. All I'll say about that for now is that they would do well to start making the design intentional.
Both the world of business and the world of software development are maturing to the point (finally) of beginning to acknowledge that the ability to perform intentional design is a critical success factor. Everything that is "To Be Done" is really "To Be Designed." And that thought process that we call design is always the underlying determiner of success -- whether we like it or not.
Who's Driving?
As a quick way to start this blog, I'm going to re-post an entry from my personal blog that's really a software development post:
---------------
In the world of software development, it's de rigeur to talk about what you're driven by. Back in the 80's, after I realized that, generally speaking, software should not just mutate like an irradiated fruit fly, I became database-driven then requirements-driven. In the early 90's, I lost myself and went through several stages of rebellion during which I was process-driven, goal-driven, architecture-driven, and finally feature-driven. Around the start of the millennium, people started telling me I should be domain-driven, test-driven, and model-driven.
An identity crisis is looming here. I'm not sure how I can be driven by all these things without coming down with some kind of software engineering schizophrenia. Who's driving this thing?
Seriously, it's clearer to me now than ever just how immature the software development field still is. I can't wait for the next new book that will proclaim to all the world that everything about software development should be [whatever]-driven. No other field takes such a simplistic view of itself. The truth of the matter is that none of these would-be umbrellas can alone drive software development. The agile development people may not always have cornered the market on best practices -- in some ways they, too, try to make software development too simple and linear -- but at least their approach is truly rational. Call it common-sense-driven if you feel you have to have a driver.
I wonder if in my lifetime I'll see the day when this field as a whole embraces the truth that software development is best modeled as a matrix of drivers that must be viewed and managed multidimensionally. And manipulated wisely. A thing does not have to be linear to be simple, and sophistication does not imply complexity.
