My data analysis bot

Posted on Dec 13, 2021

Today I spent the majority of my work day creating a massive Excel template. Tomorrow I will code the data analysis bot that will fill that template with data tables and command logs that document the analytical procedures. Over time, these two days worth of work are going to save me many, many hours.

I have always automated the data analysis part of my job, but never enough of it for my tastes. Although I am not a programmer by trade, I think like one. One of the creeds of a programmer is “don’t repeat yourself.” Sadly, at work, I repeat manual data analysis procedures all the time. Certain parts of my job resisted automation for years. Lots of work I do is ad-hoc and may never be repeated, so it rarely makes sense to automate it fully or to generalize the approach for other data files and projects. Other work I do, like pulling samples, happens all the time. I get fed up when I am performing it the same way, and no better than, how I did it years ago.

I put some though into it earlier this year and discovered that documenting the work, rather than performing it, was the biggest time-waster in my day. Documenting my work in work papers has always taken a very long time. While necessary, it is the least productive time I spend on an analysis project. (In audit, work papers are critical. They support our observations and can be followed, step-by-step, to reperform our work.)

This year, I finally figured out how to create a bot that would not only run a “standard analysis” on a certain type of dataset (that is the easy part for me), but would also document the work in an Excel work paper (that is the hard part). It is not perfect yet, but it has saved me a lot of time over the past six months. I plan to keep working on it, mostly by building new work paper templates and standard analyses, next year and into the future, so that it keeps payment me dividends.