Splunk is one of the most impressive "On Premises" log aggregation tools that I have ever come across. Being able to bring a large number of disperate data sources together into one combined index is truly useful in a modern Ops environment.

One of the things I find helpful from a development approach is consistent logging - and too often this is something that development teams overlook until things break.

However, getting data from a .NET / C# application into Splunk is not difficult and so these days I try and log absolutely everything (well, come on, the free tier gives you a decent chunk of an allowance too!).

The first thing I do is to create a new Index in Splunk - you do this by selecting Settings, Indexes and then clicking New.
The only box you need to fill in is the index name - let everything else default on your installation.

Once you have the index created, we need to setup the input. Settings then Data Inputs will take you to the right screen. Click Add New next to UDP. Pop in an unused port, say 8081, then click Next.  Make sure you select your index you created earlier, and specify the type as Generic Single Line - this basically tells Splunk it's unformatted data and not to pre-parsed it.

The next thing you need to do is actually get your code to submit data to Splunk -- the easiest way that I have found to do is to use Log4Net; in Visual Studio, install the log4net Nuget Package and this will take care of creating the relevant config entries. If, like me, you prefer to put your logging code into a common assembly then reference it elsewhere, remember to copy the assembly redirects and log4net specific entries into your other configs (or things just don't work!).

In your code, you will probably have a common class for sending log data - something like:

using log4net;
namespace YourApp.Common
{
    public static class Logging
    {
        /// <summary>
        /// Application or Class that should be identified with the log statement that is passed
        /// </summary>
        public static string Application { get; set; }
        /// <summary>
        /// Initialise logging - must be called at application start
        /// </summary>
        public static void Initialise()
        {
            log4net.Config.XmlConfigurator.Configure();
        }
        /// <summary>
        ///  Log an information message
        /// </summary>
        /// <param name="message"></param>
        public static void Info(string message)
        {
            ILog logger = LogManager.GetLogger(Application);
            logger.Info(message);
        }
    }
}

That way you can specify the application name to be passed through with the logging data (handy for Splunk, as you can throw everything into one Index and then break out specifically what you need later) - and use the class from pretty much anywhere.

In your web.config you need it to look like:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <configSections>
    <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net"/>
  </configSections>
  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
  </startup>
  <log4net>
    <appender name="UdpAppender" type="log4net.Appender.UdpAppender">
      <param name="RemoteAddress" value="splunk-server" />
      <param name="RemotePort" value="8081" />

      <layout type="log4net.Layout.PatternLayout" value="%level - %date{MM/dd HH:mm:ss} - %c - %stacktrace{2} - %message" />

    </appender>

    <root>

      <level value="ALL" />

      <appender-ref ref="UdpAppender" />
    </root>
  </log4net>
</configuration>

Finally, call away to get your data logged:



And that, folks, is it - you can now push .NET C# app log data into Splunk.

A couple of points that some people might question me on:

Why use UDP Appender and not TCP?

UDP is a lossy transmission protocol, and it is entirely possible that log messages do not make it into the Splunk indexer; however, it is significantly lighter weight than establishing TCP/IP connections.

Can I log to multiple locations - such as Splunk but also a text file?

Yes - add another Log Appender; the Log4Net docs are pretty good on this one. 

Is there much point about having the date time in the log message?

That depends - if you are worried that the messages might get cached somewhere and not always trust the date / time that Splunk adds to it's indexed entries, then you probably want to keep it. Otherwise feel free to drop it from the pattern.