Between work eating up as much time as I will let it, spending time with my girl, being somewhat social, upgrading people’s PCs, trying to get through a game, and trying to read more (currently in the middle of re-reading the Hobbit!) I have no free time. Oh yeah, I should also mention that trying to write a blog will eat up a bunch of time too.

Yeah, I want to do a lot. Probably more than I am realistically able to. However, with some work I am able to keep a surprising number of things going. Not everything I’d ever like to do, but more than I would have thought.

Some things that have helped:

• Task lists, calendars and notebooks. This probably deserves it’s own post. I’m using a bit of GTD (aka Getting Things Done) with Outlook, Google Calendar and Evernote. There are entire blogs dedicated to this that might be a better resource.
• Exercise and sleep. Yes they both take time, but you’re able to better spend the rest of the time if you stay on top of them. I’m honestly still working on this. It’s not easy!
• Getting up a bit earlier. I’m usually in bed pretty early these days, which means I sometimes get up earlier than I used to. I was never a morning person, but I do see the appeal. I’m writing this in the morning before I have to work.
• I’m spending less time on the things that don’t interest me as much. I spend far less time on Facebook now. Less time with the Xbox even than I used to. I would like to spend some free time coding again.
• If I still don’t have time to do something at some point I have to decide what to drop. I can always take it up later if I want, but it’s still not usually an easy decision.

So that’s how I’m getting by until I can make a 30 hour day or invent time travel.

So what do you think? Was this useful? Do you have any tips to add?

A little while ago I ran into some problems with the w3wp process gobbling up tons of memory. It ate up so much memory that after running for a few hours with high traffic, it would automatically get restarted by IIS.

WinDbg isn’t the friendliest looking program, but if you know a few commands you can get quite a ways with it.

In my case the memory was used up by how we were pushing NHibernate to do some non-standard things. These same steps could be used to troubleshoot most generic memory usage problems.

Here’s the steps I followed to get through this:

1. I have a memory dump I made with Process Explorer on the server when it had been running for 3 hours and was about to be recycled.
2. The Gen2 heap size was what was growing up to 1.4GB. The other heaps stayed about the same size for the life of the app. Which apparently is common when there is a memory leak. I was able to tell this with performance counters and then confirm it with windbg.
3. The type taking up the most memory is strings, after that various types of arrays. Which also seems normal, even if there weren’t a leak.
4. From a random sampling of strings, most look like SQL fragments, but some are HTML content. Makes sense, most of the strings in this particular app are going to be HTML or SQL.
5. WinDbg did take a very long time to do some operations on the 3GB memory dump. Using an SSD helped, but I still had to be patient for some operations
6. Finally it started looking like the NHibernate query cache was holding on to a bunch of data. There’s a similar issue that was discovered here.
7. The data is rooted in a static property so it will never be garbage collected. Our code was written this way since the documentation said to only create each session factory once and store it.
8. One NHibernate SessionFactory on its own isn’t too bad, it only took up 34MB. Where it gets problematic is when we have a bunch of those in memory.
9. In the near term, I think this can be fixed by using some WeakReferences. Longer term I’d like to use only one or two SessionFactories.
10. I found out that a MemoryCache would be better than WeakReferences.
    • WeakReferences go away as soon as the garbage collector runs if nobody is using them, the cached items go away when memory is tight.
    • Even longer term, I’d like to get down to just a few or even one session factory. I think reading up on sharding might be the next step here.

Links:

• SOS.dll (SOS Debugging Extension)
• Bugslayer: SOS: It’s Not Just an ABBA Song Anymore
• WinDbg / SOS Cheat Sheet
• Where’s your leak at? [Using WinDbg, SOS, and GCRoot to diagnose a .NET memory leak]
• New commands in SOS for .NET 4.0 Part 1
• New Silverlight SOS.DLL Commands: !HeapStat and !GCWhere

Useful WinDbg & SoS commands:

• .loadby sos clr – loads SoS into windbg.
  • .load C:\SomePath\sos.dll – can be used to load another version of sos.dll that was downloaded from the server.
• .prefer_dml 1 – makes addresses in the output clickable to get more details.
• CTRL+Break – stops the currently running command.
• !Help – lists all SoS commands.
• !HeapStat – lists some stats on where memory is used.
• !EEHeap -gc – lists start and end addresses for each heap.
• !DumpHeap -stat [Begin] [End] – Gives stats on what types are using up memory. Begin and End come from !EEHeap -gc, I wanted only Gen2.
• !DumpHeap -Type <Type> [Begin] [End] – Lists all instances of a type, like System.String, between Begin and End.
• !DumpObj <Address> – Prints out details on an object.
• !GCWhere <Address> – Prints out what heap an object is in, it’s address and size.
• !GCRoot -nostacks <Address> – Lists objects that have a reference to the object. The object at the root of the list is why the memory has not been garbage collected. This can take quite a while.
• !ObjSize [Address] – Lists the size of the object by adding up the size of all references. Can take a while if it needs to loop through a lot.

Details:

Here’s the output of !GCRoot 00000001805a8640 on a single string that contains a fragment of SQL. I also added some information on the total size while I was poking around.

DOMAIN(00000000001B4550):HANDLE(Pinned):73015e8:Root: 000000028010c068(System.Object[])->
00000001c0180668(System.Collections.Generic.Dictionary`2[[System.Guid, mscorlib],[NHibernate.ISessionFactory, NHibernate]])-> (1272.6 MB!)
000000020c29f578(System.Collections.Generic.Dictionary`2+Entry[[System.Guid, mscorlib],[NHibernate.ISessionFactory, NHibernate]][])-> (1272.6 MB!)
00000002402b6208(NHibernate.Impl.SessionFactoryImpl)-> (34.6 MB…)
00000002402b6628(NHibernate.Engine.Query.QueryPlanCache)->
00000002402b79c0(NHibernate.Util.SoftLimitMRUCache)->
00000002402b7a60(NHibernate.Util.LRUMap)->
00000002402b7ac0(System.Collections.Hashtable)->
00000002402b7b18(System.Collections.Hashtable+bucket[])->
00000001c08cecc8(NHibernate.Util.SequencedHashMap+Entry)->
00000001c08a6628(NHibernate.Engine.Query.HQLStringQueryPlan)->
00000001c08c9cf8(System.Object[])->
00000001c08c9c90(NHibernate.Hql.Ast.ANTLR.QueryTranslatorImpl)->
00000001c08ccc48(NHibernate.Hql.Ast.ANTLR.Tree.QueryNode)->
00000001c08caa48(NHibernate.Hql.Ast.ANTLR.Tree.FromClause)->
00000001c08cac58(NHibernate.Util.NullableDictionary`2[[System.String, mscorlib],[NHibernate.Hql.Ast.ANTLR.Tree.FromElement, NHibernate]])->
00000001c08cac80(System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[NHibernate.Hql.Ast.ANTLR.Tree.FromElement, NHibernate]])->
00000001c08cb748(System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[NHibernate.Hql.Ast.ANTLR.Tree.FromElement, NHibernate]][])->
00000001c08cb538(NHibernate.Hql.Ast.ANTLR.Tree.FromElement)->
00000001c08cb650(NHibernate.Hql.Ast.ANTLR.Tree.FromElementType)->
0000000180481ef0(NHibernate.Persister.Collection.BasicCollectionPersister)->
00000001805a83f0(NHibernate.Loader.Collection.BasicCollectionLoader)-> (34.6 MB…)
00000001805a87a0(NHibernate.SqlCommand.SqlString)-> (616 Bytes…)
00000001805a8740(System.Object[])->
00000001805a8640(System.String) (200 Bytes…)

• Command line tools: logman.exe, relog.exe and typeperf.exe.
• Logman can be used to create new logs. It has one option that the GUI doesn’t have, and that is to run the same scheduled log each day.
• Relog can be used to convert log files from the binary .blg into .csv or SQL. It can also let you pull only specific counters or time spans out of a .blg. Another use is combining counter logs from multiple files.
• Typeperf can be used to get raw counter data from the command line.
• There are some commands in PowerShell too, but for adding counters logman seems much less verbose.
• I have some productions servers logging from early morning until night and the files are only around 15MB or so.
• They’re only logging selected counters and only logging data once a minute.
• Some counter logs can get large very quickly because they grab a lot of data every few seconds, but the file size seems to be capped around 3GB.
• Here is the list of counters I’m currently capturing. I’d love to hear any suggestions if there are useful ones I missed.
\.NET CLR Exceptions(_Global_)\# of Exceps Thrown / sec
\.NET CLR Interop(_Global_)\# of Stubs
\.NET CLR Loading(_Global_)\Bytes in Loader Heap
\.NET CLR Loading(_Global_)\Current Assemblies
\.NET CLR Loading(_Global_)\Rate of Assemblies
\.NET CLR Memory(_Global_)\# Bytes in all Heaps
\.NET CLR Memory(_Global_)\# Induced GC
\.NET CLR Memory(_Global_)\% Time in GC
\.NET CLR Memory(_Global_)\Gen 0 heap size
\.NET CLR Memory(_Global_)\Gen 1 heap size
\.NET CLR Memory(_Global_)\Gen 2 heap size
\.NET CLR Memory(_Global_)\Large Object Heap size
\ASP.NET\Application Restarts
\ASP.NET\Request Execution Time
\ASP.NET\Request Wait Time
\ASP.NET\Requests Current
\ASP.NET\Worker Process Restarts
\ASP.NET Applications(__Total__)\Compilations Total
\ASP.NET Applications(__Total__)\Errors During Execution
\ASP.NET Applications(__Total__)\Request Execution Time
\ASP.NET Applications(__Total__)\Request Wait Time
\Memory\% Committed Bytes In Use
\Memory\Available MBytes
\Memory\Pages/sec
\PhysicalDisk(_Total)\Avg. Disk sec/Read
\PhysicalDisk(_Total)\Avg. Disk sec/Write
\Process(_Total)\Handle Count
\Process(_Total)\Virtual Bytes
\Processor(_Total)\% Processor Time
\Web Service(_Total)\Bytes Received/sec
\Web Service(_Total)\Bytes Sent/sec
\Web Service(_Total)\Get Requests/sec
\Web Service(_Total)\Post Requests/sec

• There are a couple buttons in the toolbars for copying and pasting settings.
• These can be used to paste the settings into notepad and edit it as XML.
• It was useful for using the same settings on multiple servers as well as getting the names of counters for the command line.
• The Windows 7 Performance Monitor has a Scale Selected Counter option in the right click menu that will automatically change the scale setting so that the graph fits in the window.

Examples

As an example, here’s a couple graphs of a server I was testing a while ago. First, traffic to the server. It doesn’t look like it was all that busy, almost 5 requests a second on average:

And here’s memory usage, you can see how the Gen 2 heap size grows and grows up to around 1.5GB when the app pool recycles:

It took me a while, but I finally merged the content from my old blog to this one. It wasn’t too hard since they’re both WordPress. Just export and then import. The parts that took the longest were finding a way to exclude the old posts from the RSS feed and setting up permanent redirects for all the posts and feeds.

Hopefully the transition goes smoothly from here on out. I’ll be watching my logs, stats, and webmaster tools over the next few days to make sure.

So now I need to get to writing some real content. There’s a lot of old (and probably outdated) content here now and I need to get it more up to date.

That’s a good chunk of the reason I started my original blog: to get practice writing. I figured I’d never get better without practice, and practicing in public certainly makes me more careful about what I write and how sloppy I let my writing get.

I’ve been writing emails more since I’m not getting as much face-to-face communication. I’ve noticed it’s almost more about what I don’t write than what I do write. I’ve written giant wall of text emails before, and I think that caused people to just skim it. It takes a lot of work to write a good email that will get your ideas across and get people to respond with what you want.

I’m no designer and I know I could find someone more talented to do this kind of thing, but this isn’t exactly a commercial site. I’ll probably keep tweaking the look a bit to make it a little better, but I want to stick with something that looks kind of like this.

Someday soon, I’m going to try to move my old content over to this blog.

Then I’ll have to actually spend time writing some content instead of distracting myself with technical things to tweak.

Oh, and the redesign looks decent on a mobile browser. Or at least it’s pretty usable.

At first I tried sitting over on the side of the room near the window and raising my laptop up higher to get a better signal. Even with that, the signal was still barely usable. It might have worked for sending an email or loading a single web page, but I needed to get some real work done.

After fighting with it for a while, it occurred to me that I had an old wifi router with me. A good old Linksys WRT54-GL.

Originally I had Tomato on it, so I connected it to my laptop with an ethernet cable and got to work. I was able to get it to find the network and connect. Now I could put the router in the top corner of the window where the signal was best and run a wire over to my laptop.

Being hardwired worked, but it still wasn’t ideal. So I got to wondering if I could simply extend the range of the network. To make a long story short, I couldn’t figure out how to do it with Tomato. There is a way, but you need to setup both routers to get it to work.

So I installed DD-WRT. It has a feature I’ve never seen on any other router: it supports virtual wireless interfaces. This meant I could have it connect to the public wifi while at the same time being another access point.

So I set it up to repeater mode, gave it everything it needed to connect to the wifi signal, and added a virtual interface:

Before I was done, I wanted to go and setup some security. It was nice I could do this separately for each interface:

After doing all this and rebooting a few times, it worked! I had a router that would essentially use a hotspot as a wan port. It was a little finicky at first, but once it started working it was rock solid.

I was still able to setup my own network, run a DHCP server, and do everything else like normal. The router would NAT everything to the hotspot as if it were my ISP. I’m assuming that the hotspot was also running it’s own NAT as well, but that didn’t matter to me.

After I got all that to work, I setup a VPN and tunneled all my traffic through it. Just in case someone malicious was also using the hotspot. It also opened up a lot of possibilities for me like forwarding a port on a public IP back to me or someday running IPv6.

We’re using loadbalancer.org. Which uses Linux-HA which uses ldirectord to monitor servers.

There are a few options for how it tests to see if a server is down. We had been using connect. What that does is it just opens a TCP connection on port 80. If it opens successfully, it assumes the server is alive. The problem with that was that the app pool was stopped, but other app pools were still running so IIS was still accepting TCP connections.

What I want to do is change it to negotiate. What negotiate does is it sends an HTTP request and expects to get specific content back.

I tried negotiate last week on QA servers, and it immediately marked every server as unavailable. Not a good sign. What I finally realized is that it’s sending the HTTP request to the server’s primary IP and not the load balanced IP. As soon as I made the IIS web accept connections on that IP, it worked as advertised.

I’ve setup the web on our QA servers to accept connections on port 3333 with a specific host header. That way the load balancer can check if the app pool is running and the file can be stored in our project, but it’s unlikely someone will accidentally browse to it. (I’m thinking about reversing this decision for production honestly.)

After I made the changes to QA, I tried stopping app pools and causing crashes by changing the version of asp.net used. I never got any errors in my browser and the server was marked as non-nactive.

This could use a little more testing, but I think it’s a good solution.

Unfortunately it’s pretty poor at handling a few situations: when several different threads have the same subject and when the subject line is changed. In theory, there should be a way for Outlook to work around those limitations using standard emails headers, but it doesn’t. Maybe in 2012…

Recently I had several different email threads going at once with tens of emails a day for the same project. Unfortunately one of the participants had a spam filter that would append [!!SPAM] to nearly every subject line. If I didn’t correct the subject, we’d eventually get a subject like RE: [!!SPAM] Re: [!!SPAM] RE: You get the idea.

While you can change the subject of an email in your inbox, it’s still grouped into a separate conversation.

So I went looking and eventually found a tool called MFCMAPI that will let you poke into all kinds of hidden fields in Outlook. Make sure you get the 64bit version if you’re using the 64bit Outlook.

After starting a session and connecting, it gave me a list of mailboxes:

Double clicking my mailbox brought up this:

And double clicking on my inbox brought up a list of emails. If I double clicked on an email, it would open up in Outlook. However, just selecting the email would show a list of properties below:

I ended up editing PR_CONVERSATION_TOPIC and PR_SUBJECT:

The PR_CONVERSATION_TOPIC field is used by Outlook to group emails into the same conversation. As far as I can tell it can be nearly any unique string. I just changed it to remove the [!!SPAM] and the RE: since that’s what other messages already in the conversation used.

This has helped me quite a bit to get organized when I was being overwhelmed with emails. It’s probably not foolproof, so be careful what you change. I only changed the two fields and I’m not even sure what half the other ones are.

Now that I know which fields to change I was hoping to find or make a plugin that would do this from within Outlook automatically.

A few thoughts to throw on the pile of comments already there:

I like semantic versioning like 4.1.2, but I could see that maybe the marketing department would want to make the name a little friendlier. If some words are in there, keep it simple and consistent. Wiz Bang 2011 SP 2 Beta 3 is the most complex I should ever have to parse. And that’s if I want to get all technical and play with pre-release software.

I think the most a non-developer, like your boss or manager, should really care about is two numbers. One for the version of the product, Office 2010 for example, and one for updates of at least moderate size that increments the SP or .x number.

Lose the acronyms. How many people know what CTP or PU or even R stand for? At least stick with SP like Windows had been doing for years. Couldn’t CTP be replaced with the word Beta, or Preview?

Put the numbers in descending order of importance. If it’s going to be Wiz Bang 2011, fine, I’ll treat 2011 as the major version that I am using. Wiz Bang 2011 SP1 would be alright too if it’s a collection of medium to small changes. If it’s a new, incompatible version that’s being sold separately, don’t add an R2, change the major number from 2008 to 2010.

And finally if you’re already using decimal notation, like 3.5, don’t throw other numbers around. Update that number or tack another .1 on there. 3.5.1 means it’s a little bug fix update. 3.5 R2 means “say what?”