Here’s four things I try to do that help:

• I try to keep my inbox empty. See http://inboxzero.com/.
• I try not to check my email all the time. This is hard for me.
• I try to make a note of anything that’s nagging at the back of my mind while I’m trying to focus.
• I drink better coffee now that I’m much better at making it.

They don’t make things perfect, but at least I feel like I’m in control now.

Email

I lose a lot of time to things like emails coming in, IM conversations. Just one email or IM discussion can easily eat an hour. I really try to be as helpful as I can, but sometimes I don’t have time left at the end of the day.

I’ve been pretty good about implementing Inbox Zero. I rarely have any emails left in my inbox by the end of the day and I have filters for mailing lists and CC’s. However I’d like to start processing my email in batches, so I don’t have to deal with it first thing in the day. The problem is that as soon as Outlook opens and I glance at that number, I feel compelled to take a few minutes and at least sort them. And then if there’s just one that doesn’t make sense or asks me to do something I can’t get done, it nags at me the rest of the day.

Notes

Another thing that’s helping a lot is just writing down anything I can’t get to right at the moment. Sometimes I’ll be debugging some problem and I’ll notice code that could be improved, or I get an idea that would make error logging easier, or whatever. Rather than trying to make the change right away and get sidetracked, I used to try to remember that until I got done. The problem was that I would either forget about it or I’d keep thinking about it when I needed to focus on other things.

Now I just create a one line note in Evernote. I can move it and tag it later when I have time. Or sometimes I’ll just delete it outright because it doesn’t seem like such a hot idea anymore.

I’d originally done this with many SessionFactories (like this post does.)

After a while I ran into memory usage issues (See this post about debugging with WinDbg.)

So that’s when I started looking for another option. What I wanted to do was use one factory and supply it with connections to the right databases.

The Provider

/// <summary>
/// Connection provider that will override the default connection string and use one that's
/// been specified for this thread.
/// </summary>
public class DbSpecificConnectionProvider : DriverConnectionProvider
{
    /// <summary>
    /// Returns a new connection using the connection string set in ThreadConnectionString.
    /// </summary>
    public override IDbConnection GetConnection()
    {
        var connection = Driver.CreateConnection();
        try
        {
            connection.ConnectionString = GetConnectionStringFromContext();
            connection.Open();
        }
        catch (DbException)
        {
            connection.Dispose();
            throw;
        }
        return connection;
    }
}

This isn’t too complex really. It inherits from the default provider of DriverConnectionProvider and overrides the GetConnection function. It then gets a connection from the driver, sets the connection string and opens the connection. If there are any errors it’ll cleanup the connection before exiting.

Additional bits

You’ll also need to edit your configuration to tell it about the new provider:

<property name="connection.provider">
  My.Namespace.DbSpecificConnectionProvider, MyAssembly
</property>

And if you are using second level caching, you might need to implement a caching provider so that the cache doesn’t get cross-contaminated.

Now this hasn’t had that much testing yet, so I’m still being a little cautious with it. But if it works, it’s a nice clean solution to my specific problem.

Between work eating up as much time as I will let it, spending time with my girl, being somewhat social, upgrading people’s PCs, trying to get through a game, and trying to read more (currently in the middle of re-reading the Hobbit!) I have no free time. Oh yeah, I should also mention that trying to write a blog will eat up a bunch of time too.

Yeah, I want to do a lot. Probably more than I am realistically able to. However, with some work I am able to keep a surprising number of things going. Not everything I’d ever like to do, but more than I would have thought.

Some things that have helped:

• Task lists, calendars and notebooks. This probably deserves it’s own post. I’m using a bit of GTD (aka Getting Things Done) with Outlook, Google Calendar and Evernote. There are entire blogs dedicated to this that might be a better resource.
• Exercise and sleep. Yes they both take time, but you’re able to better spend the rest of the time if you stay on top of them. I’m honestly still working on this. It’s not easy!
• Getting up a bit earlier. I’m usually in bed pretty early these days, which means I sometimes get up earlier than I used to. I was never a morning person, but I do see the appeal. I’m writing this in the morning before I have to work.
• I’m spending less time on the things that don’t interest me as much. I spend far less time on Facebook now. Less time with the Xbox even than I used to. I would like to spend some free time coding again.
• If I still don’t have time to do something at some point I have to decide what to drop. I can always take it up later if I want, but it’s still not usually an easy decision.

So that’s how I’m getting by until I can make a 30 hour day or invent time travel.

So what do you think? Was this useful? Do you have any tips to add?

A little while ago I ran into some problems with the w3wp process gobbling up tons of memory. It ate up so much memory that after running for a few hours with high traffic, it would automatically get restarted by IIS.

WinDbg isn’t the friendliest looking program, but if you know a few commands you can get quite a ways with it.

In my case the memory was used up by how we were pushing NHibernate to do some non-standard things. These same steps could be used to troubleshoot most generic memory usage problems.

Here’s the steps I followed to get through this:

1. I have a memory dump I made with Process Explorer on the server when it had been running for 3 hours and was about to be recycled.
2. The Gen2 heap size was what was growing up to 1.4GB. The other heaps stayed about the same size for the life of the app. Which apparently is common when there is a memory leak. I was able to tell this with performance counters and then confirm it with windbg.
3. The type taking up the most memory is strings, after that various types of arrays. Which also seems normal, even if there weren’t a leak.
4. From a random sampling of strings, most look like SQL fragments, but some are HTML content. Makes sense, most of the strings in this particular app are going to be HTML or SQL.
5. WinDbg did take a very long time to do some operations on the 3GB memory dump. Using an SSD helped, but I still had to be patient for some operations
6. Finally it started looking like the NHibernate query cache was holding on to a bunch of data. There’s a similar issue that was discovered here.
7. The data is rooted in a static property so it will never be garbage collected. Our code was written this way since the documentation said to only create each session factory once and store it.
8. One NHibernate SessionFactory on its own isn’t too bad, it only took up 34MB. Where it gets problematic is when we have a bunch of those in memory.
9. In the near term, I think this can be fixed by using some WeakReferences. Longer term I’d like to use only one or two SessionFactories.
10. I found out that a MemoryCache would be better than WeakReferences.
    • WeakReferences go away as soon as the garbage collector runs if nobody is using them, the cached items go away when memory is tight.
    • Even longer term, I’d like to get down to just a few or even one session factory. I think reading up on sharding might be the next step here.

Links:

• SOS.dll (SOS Debugging Extension)
• Bugslayer: SOS: It’s Not Just an ABBA Song Anymore
• WinDbg / SOS Cheat Sheet
• Where’s your leak at? [Using WinDbg, SOS, and GCRoot to diagnose a .NET memory leak]
• New commands in SOS for .NET 4.0 Part 1
• New Silverlight SOS.DLL Commands: !HeapStat and !GCWhere

Useful WinDbg & SoS commands:

• .loadby sos clr – loads SoS into windbg.
  • .load C:\SomePath\sos.dll – can be used to load another version of sos.dll that was downloaded from the server.
• .prefer_dml 1 – makes addresses in the output clickable to get more details.
• CTRL+Break – stops the currently running command.
• !Help – lists all SoS commands.
• !HeapStat – lists some stats on where memory is used.
• !EEHeap -gc – lists start and end addresses for each heap.
• !DumpHeap -stat [Begin] [End] – Gives stats on what types are using up memory. Begin and End come from !EEHeap -gc, I wanted only Gen2.
• !DumpHeap -Type <Type> [Begin] [End] – Lists all instances of a type, like System.String, between Begin and End.
• !DumpObj <Address> – Prints out details on an object.
• !GCWhere <Address> – Prints out what heap an object is in, it’s address and size.
• !GCRoot -nostacks <Address> – Lists objects that have a reference to the object. The object at the root of the list is why the memory has not been garbage collected. This can take quite a while.
• !ObjSize [Address] – Lists the size of the object by adding up the size of all references. Can take a while if it needs to loop through a lot.

Details:

Here’s the output of !GCRoot 00000001805a8640 on a single string that contains a fragment of SQL. I also added some information on the total size while I was poking around.

DOMAIN(00000000001B4550):HANDLE(Pinned):73015e8:Root: 000000028010c068(System.Object[])->
00000001c0180668(System.Collections.Generic.Dictionary`2[[System.Guid, mscorlib],[NHibernate.ISessionFactory, NHibernate]])-> (1272.6 MB!)
000000020c29f578(System.Collections.Generic.Dictionary`2+Entry[[System.Guid, mscorlib],[NHibernate.ISessionFactory, NHibernate]][])-> (1272.6 MB!)
00000002402b6208(NHibernate.Impl.SessionFactoryImpl)-> (34.6 MB…)
00000002402b6628(NHibernate.Engine.Query.QueryPlanCache)->
00000002402b79c0(NHibernate.Util.SoftLimitMRUCache)->
00000002402b7a60(NHibernate.Util.LRUMap)->
00000002402b7ac0(System.Collections.Hashtable)->
00000002402b7b18(System.Collections.Hashtable+bucket[])->
00000001c08cecc8(NHibernate.Util.SequencedHashMap+Entry)->
00000001c08a6628(NHibernate.Engine.Query.HQLStringQueryPlan)->
00000001c08c9cf8(System.Object[])->
00000001c08c9c90(NHibernate.Hql.Ast.ANTLR.QueryTranslatorImpl)->
00000001c08ccc48(NHibernate.Hql.Ast.ANTLR.Tree.QueryNode)->
00000001c08caa48(NHibernate.Hql.Ast.ANTLR.Tree.FromClause)->
00000001c08cac58(NHibernate.Util.NullableDictionary`2[[System.String, mscorlib],[NHibernate.Hql.Ast.ANTLR.Tree.FromElement, NHibernate]])->
00000001c08cac80(System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[NHibernate.Hql.Ast.ANTLR.Tree.FromElement, NHibernate]])->
00000001c08cb748(System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[NHibernate.Hql.Ast.ANTLR.Tree.FromElement, NHibernate]][])->
00000001c08cb538(NHibernate.Hql.Ast.ANTLR.Tree.FromElement)->
00000001c08cb650(NHibernate.Hql.Ast.ANTLR.Tree.FromElementType)->
0000000180481ef0(NHibernate.Persister.Collection.BasicCollectionPersister)->
00000001805a83f0(NHibernate.Loader.Collection.BasicCollectionLoader)-> (34.6 MB…)
00000001805a87a0(NHibernate.SqlCommand.SqlString)-> (616 Bytes…)
00000001805a8740(System.Object[])->
00000001805a8640(System.String) (200 Bytes…)

• Command line tools: logman.exe, relog.exe and typeperf.exe.
• Logman can be used to create new logs. It has one option that the GUI doesn’t have, and that is to run the same scheduled log each day.
• Relog can be used to convert log files from the binary .blg into .csv or SQL. It can also let you pull only specific counters or time spans out of a .blg. Another use is combining counter logs from multiple files.
• Typeperf can be used to get raw counter data from the command line.
• There are some commands in PowerShell too, but for adding counters logman seems much less verbose.
• I have some productions servers logging from early morning until night and the files are only around 15MB or so.
• They’re only logging selected counters and only logging data once a minute.
• Some counter logs can get large very quickly because they grab a lot of data every few seconds, but the file size seems to be capped around 3GB.
• Here is the list of counters I’m currently capturing. I’d love to hear any suggestions if there are useful ones I missed.
\.NET CLR Exceptions(_Global_)\# of Exceps Thrown / sec
\.NET CLR Interop(_Global_)\# of Stubs
\.NET CLR Loading(_Global_)\Bytes in Loader Heap
\.NET CLR Loading(_Global_)\Current Assemblies
\.NET CLR Loading(_Global_)\Rate of Assemblies
\.NET CLR Memory(_Global_)\# Bytes in all Heaps
\.NET CLR Memory(_Global_)\# Induced GC
\.NET CLR Memory(_Global_)\% Time in GC
\.NET CLR Memory(_Global_)\Gen 0 heap size
\.NET CLR Memory(_Global_)\Gen 1 heap size
\.NET CLR Memory(_Global_)\Gen 2 heap size
\.NET CLR Memory(_Global_)\Large Object Heap size
\ASP.NET\Application Restarts
\ASP.NET\Request Execution Time
\ASP.NET\Request Wait Time
\ASP.NET\Requests Current
\ASP.NET\Worker Process Restarts
\ASP.NET Applications(__Total__)\Compilations Total
\ASP.NET Applications(__Total__)\Errors During Execution
\ASP.NET Applications(__Total__)\Request Execution Time
\ASP.NET Applications(__Total__)\Request Wait Time
\Memory\% Committed Bytes In Use
\Memory\Available MBytes
\Memory\Pages/sec
\PhysicalDisk(_Total)\Avg. Disk sec/Read
\PhysicalDisk(_Total)\Avg. Disk sec/Write
\Process(_Total)\Handle Count
\Process(_Total)\Virtual Bytes
\Processor(_Total)\% Processor Time
\Web Service(_Total)\Bytes Received/sec
\Web Service(_Total)\Bytes Sent/sec
\Web Service(_Total)\Get Requests/sec
\Web Service(_Total)\Post Requests/sec

• There are a couple buttons in the toolbars for copying and pasting settings.
• These can be used to paste the settings into notepad and edit it as XML.
• It was useful for using the same settings on multiple servers as well as getting the names of counters for the command line.
• The Windows 7 Performance Monitor has a Scale Selected Counter option in the right click menu that will automatically change the scale setting so that the graph fits in the window.

Examples

As an example, here’s a couple graphs of a server I was testing a while ago. First, traffic to the server. It doesn’t look like it was all that busy, almost 5 requests a second on average:

And here’s memory usage, you can see how the Gen 2 heap size grows and grows up to around 1.5GB when the app pool recycles:

It took me a while, but I finally merged the content from my old blog to this one. It wasn’t too hard since they’re both WordPress. Just export and then import. The parts that took the longest were finding a way to exclude the old posts from the RSS feed and setting up permanent redirects for all the posts and feeds.

Hopefully the transition goes smoothly from here on out. I’ll be watching my logs, stats, and webmaster tools over the next few days to make sure.

So now I need to get to writing some real content. There’s a lot of old (and probably outdated) content here now and I need to get it more up to date.

That’s a good chunk of the reason I started my original blog: to get practice writing. I figured I’d never get better without practice, and practicing in public certainly makes me more careful about what I write and how sloppy I let my writing get.

I’ve been writing emails more since I’m not getting as much face-to-face communication. I’ve noticed it’s almost more about what I don’t write than what I do write. I’ve written giant wall of text emails before, and I think that caused people to just skim it. It takes a lot of work to write a good email that will get your ideas across and get people to respond with what you want.

I’m no designer and I know I could find someone more talented to do this kind of thing, but this isn’t exactly a commercial site. I’ll probably keep tweaking the look a bit to make it a little better, but I want to stick with something that looks kind of like this.

Someday soon, I’m going to try to move my old content over to this blog.

Then I’ll have to actually spend time writing some content instead of distracting myself with technical things to tweak.

Oh, and the redesign looks decent on a mobile browser. Or at least it’s pretty usable.

At first I tried sitting over on the side of the room near the window and raising my laptop up higher to get a better signal. Even with that, the signal was still barely usable. It might have worked for sending an email or loading a single web page, but I needed to get some real work done.

After fighting with it for a while, it occurred to me that I had an old wifi router with me. A good old Linksys WRT54-GL.

Originally I had Tomato on it, so I connected it to my laptop with an ethernet cable and got to work. I was able to get it to find the network and connect. Now I could put the router in the top corner of the window where the signal was best and run a wire over to my laptop.

Being hardwired worked, but it still wasn’t ideal. So I got to wondering if I could simply extend the range of the network. To make a long story short, I couldn’t figure out how to do it with Tomato. There is a way, but you need to setup both routers to get it to work.

So I installed DD-WRT. It has a feature I’ve never seen on any other router: it supports virtual wireless interfaces. This meant I could have it connect to the public wifi while at the same time being another access point.

So I set it up to repeater mode, gave it everything it needed to connect to the wifi signal, and added a virtual interface:

Before I was done, I wanted to go and setup some security. It was nice I could do this separately for each interface:

After doing all this and rebooting a few times, it worked! I had a router that would essentially use a hotspot as a wan port. It was a little finicky at first, but once it started working it was rock solid.

I was still able to setup my own network, run a DHCP server, and do everything else like normal. The router would NAT everything to the hotspot as if it were my ISP. I’m assuming that the hotspot was also running it’s own NAT as well, but that didn’t matter to me.

After I got all that to work, I setup a VPN and tunneled all my traffic through it. Just in case someone malicious was also using the hotspot. It also opened up a lot of possibilities for me like forwarding a port on a public IP back to me or someday running IPv6.

We’re using loadbalancer.org. Which uses Linux-HA which uses ldirectord to monitor servers.

There are a few options for how it tests to see if a server is down. We had been using connect. What that does is it just opens a TCP connection on port 80. If it opens successfully, it assumes the server is alive. The problem with that was that the app pool was stopped, but other app pools were still running so IIS was still accepting TCP connections.

What I want to do is change it to negotiate. What negotiate does is it sends an HTTP request and expects to get specific content back.

I tried negotiate last week on QA servers, and it immediately marked every server as unavailable. Not a good sign. What I finally realized is that it’s sending the HTTP request to the server’s primary IP and not the load balanced IP. As soon as I made the IIS web accept connections on that IP, it worked as advertised.

I’ve setup the web on our QA servers to accept connections on port 3333 with a specific host header. That way the load balancer can check if the app pool is running and the file can be stored in our project, but it’s unlikely someone will accidentally browse to it. (I’m thinking about reversing this decision for production honestly.)

After I made the changes to QA, I tried stopping app pools and causing crashes by changing the version of asp.net used. I never got any errors in my browser and the server was marked as non-nactive.

This could use a little more testing, but I think it’s a good solution.