Is there no good and fast way to create an ETL job in Python?

I love using PETL to read, transform, and validate datasets. Its API is simple and straightforward to use, and its documentation, while not perfect, is really good. Unfortunately, PETL’s data processing—especially loading data into SQL Server—is very slow—to the point of being completely unusable for anything but the smallest datasets.

Odo promises to be much faster at data loads, and seems dead-simple to use, but I cannot get it to work. It turns out that the package hasn’t been updated in five years, which is why it will not work with the modern Python version I am using. I just wasted an hour on it and will probably give it up completely.

I can’t find any other ETL tools that will do the data load for me. I have to rely on the SQL Server Import/Export Wizard, which usually works and is fast but is a one-shot thing because I don’t have SQL Server Integration Services at my disposal.

I suppose I will either have to fork Odo, get it working, and figure out how to get my fork of Odo into my project, or I will have to write my own import routine using SQL Alchemy. Both seem difficult to me and not worth the trouble. I’m not the database administrator on my project—I’m supposed to be the data analyst!

Long-running data feeds are a real thing after all

Over the weekend I wrote a SQL Server stored procedure and some user-defined functions it relies on that cleaned up data and inserted it into a new table. It was the sort of thing that took a short time to write, but an awfully long time to execute and to troubleshoot. It gave me an opportunity to learn more about using SQL Server’s “Include Actual Execution Plan” and “Include Live Query Statistics” features, which helped point me in the direction of creating an index that supposedly sped the whole process up by 30%.

The first version of my load script ran for 9.5 hours, and then bombed because it tried to insert a NULL value into the primary key of a table. Zero rows were inserted. I revised it to create a second version, which eventually got killed by the server (it timed out or over-burdened the database server, I guess) before it completed after over ten hours. Zero rows were inserted. The final version ran yesterday and overnight last night for over 16 hours, which makes it the longest-running data feed I have ever written by over 10 hours. This run was successful. Only 37,000 rows were inserted, but to get the data I had to comb over 45 million records, which is why it took so long.

As an auditor, I have not always believed it when IT would tell me a data feed took all weekend to run. This experience helped me understand than it can happen, especially when it is an ad-hoc job.

My current Python project

I have been burning the midnight oil over the past week writing a data validator program—basically a front-end to the PETL validate function—in Python. This would be my third such program, and it is going to be the sort where it can be configured entirely, and flexibly, with configuration files. Theoretically, it should not require coding, or at least not that much coding, to set up validation rules. It has been a fun project thus far, and it is nearing the point where all the pieces will come together.

Lately I have been writing unit tests, and that has been where all the pain points have been. I used to use Python’s standard unittest framework for unit testing, but it requires so much repetitive code that I decided to use pytest instead. Pytest lets you parametrize unit test functions, which reduces code repetition considerably.

I never used pytest before, and ended up spending a lot of time trying to figure out how to get it working correctly. Getting my test module access to the package I’m testing was the first major hurdle. After that I spent a couple hours figuring out that I couldn’t make test fixtures that return lists, and that I couldn’t use text fixtures as test parameters. For the latter, I found a workaround in the form of the pytest-lazy-fixture package.

Generating dummy data with Python

Today I learned a good way to generate dummy data for use in the data analysis training I am going to perform next month.

While there are services like dumbdata.com that can produce dummy datasets without requiring any programming at all, those tend not to work for me. My data needs are domain-specific. I don’t just need random names and addresses and things like that; I also need specific columns, including some inter-related financial data columns, for a dummy dataset to make sense to my audience.

I have been using Python a lot lately, so naturally I wrote a Python script to generate a table full of randomized, but real-looking, data. First I used petl’s dummytable command to create a base table full of randomized identifier numbers, dates, data categories, and dollar amounts. To generate real-looking data for that table, I used functionality from the Faker package and from the standard random library, including Faker.date_between and random.choice. Then I used petl’s addfield to add some fields with calculated and inter-related values. Next, I used the petl cut function to re-order a subset of the table columns and prepare them for export. Lastly, I used the petl toxlsx function to export the data to Excel.

It was surprisingly easy. Not having to write any of the functions to randomly generate the data or pick random selections from value lists made the process far quicker than it otherwise would have been. I wish I had known about these tools the last time I created a data analytics training demo.

A Python Refactoring Day

I have spent practically all day coding, coding, coding. I am adding features to, and (more than anything) refactoring, the Python data validation program that I have been working on for my day job. The result isn’t perfect, but it is far better than where it was this morning.

First, I broke up my 600-line script into numerous module files. I was happy to learn how easy this is, but I am still a novice at what I can do with the Python package I have created. Having multiple files allowed me to refactor the major features of the application, and to identify the customization points I need to work on next, without getting lost in a single, enormous source code file.

Then, I started to delete stale code, refactor old but working code, and optimize the performance of the major data pipelines. It felt good to make everything streamlined.

Lastly, I did some testing. I don’t have proper unit tests, and still need to learn how to structure a Python package properly to allow for unit testing. That may be tomorrow’s task.

Refactoring Python

I completed my Python program this morning, but could not resist refactoring it this evening. I cut my main method’s size in half, compartmentalized my code better in classes, and made the program structure more data-driven rather than process-driven.

The work should pay off for me, because I will have to make copies of my program and customize each copy for a different type of data valuation.

I probably over-engineered it, though; it went from 525 lines to 625 in the process.

Python for data validation

I spent a good chunk of time today creating a data validation program using Python. Python is a language I know (to some extent) but I barely use. Instead, I write scripts in PowerShell, create more complex programs in C#, and analyze data with SQL queries, Galvanize Analytics, or Power Query in Excel. My goal today was to find a way to validate a large number of .csv files, which have a great many columns and will be of questionable quality. My normal tools and languages would be either too cumbersome or too limited to do this, so I reached for Python.

Python is great at importing .csv files, parsing them, modifying them, and outputting a modified copy. Within a few minutes of research, I discovered a Python package called petl, which contains a ready-made data validation pipeline. I just needed to create validation functions, which are simple 1-3 line functions, and use them to define constraints, which are simply dictionaries. All those constraints get put into an array and passed to an already-written validation method.

Coding this program has been fun and remarkably efficient. It has been fun coding in Python again. I particularly love how packages can make the hard parts easy and leave me with more time to spend on my data work rather than on creating scaffolding for the program. I will have to look into more uses for Python going forward.

I published a Scriptable script that lets you publish text posts to Micro.blog from any iOS text editor using the standard share sheet. I am excited to share it with the community. Maybe @jean or @manton would be kind enough to share it more widely than I can.

I am working on a Scriptable script to publish to Micro.blog via iOS share sheets, so I don’t have to rely on Drafts for that purpose forever. I am preparing it for publication on GitHub, and am making too many changes to its variable names right now in the process. 😅

I’m back to late-night debugging: a nasty bug has gotten worse, not better, since I started compiling my app on the iOS 14 SDK. It’s not the best idea to wait until late at night to think really hard about a mind-bending problem, but that’s the only time I have to myself.

I am still struggling to get my app rewrite done in UIKit. Next year, for real this time, I’m going to learn Swift UI.

I have been using Xcode for so long and I just now realized that Control+Command+Left/Right Arrow goes back and forward in the editor’s navigation stack. Is there a head-smack emoji? 😅

Does Xcode 12 beta 1 crash less than Xcode 11.5?

Since I started dogfooding my new Mac app, I have found a few new bugs that I never noticed before. That “having never noticed it before” feeling is so weird sometimes that it stops me in my tracks.

SwiftoDo Development Notes, September 2018

Files integration

In late August I released a version of SwiftoDo that added sync support for any cloud data provider, via integration with the Files app. I think that this integration can be improved in the future. For example, right now, you cannot create a todo.txt file in the Files app using SwiftoDo, and you cannot open an existing todo.txt from the Files app.

If I add those features, I may as well rewrite the app’s UI, so that you have to create or open a file upon SwiftoDo’s launch. I would also have to rewrite how preferences are stored, so users could define different preferences for different files they open.

Those changes would, I think, necessitate dropping the offline support features that currently exist—namely manual sync mode, and the failsafes in place for when automatic sync fails (typically due to network unavailability). I am actually not sure how other document-based apps on iOS handle things when network connectivity is lost or unavailable. I would assume they simply cannot work without a constant network connection, because they cannot access their file, but I am not sure.

I do know that a task list is not a typical document-based app, like a text editor. Users typically want it to be always available, rather than dependent on a constantly-available network connection. Because I would rather not remove offline features from my app, and because I currently have very little time for app development, I do not plan any big changes to the app related to Files integration in the near future. When they do happen, I would expect that the UI of the app would be changed pretty significantly.

iOS 12 Support for SwiftoDo

The current version of SwiftoDo runs on iOS 12 without incident. I have, however, compiled a new version of SwiftoDo on the iOS 12 SDK. It has no new features, but the SDK is newer, so it inherits upstream bug fixes from Apple, and I updated my codebase to Swift 4.2. I am dropping support for iOS 10.x, too, because iOS 12 will be released imminently, and it honestly makes no sense for anyone with an iOS device from the past four years or so not to upgrade to it (iOS 12 performance is that good). Because of the under-the-hood changes, and the dropped compatibility with iOS 10, I am bumping the version number to 3.0.0. (Don’t get too excited!)

macOS Mojave Support for SwiftoDo Desktop

I am running the MacOS Mojave beta, and have been testing out its new dark mode. I love dark mode, and it took me about two seconds to realize that dark mode support is not a nice-to-have—it is an absolute necessity. Therefore, have coded support for it in a new build of SwiftoDo Desktop. I will submit it to the App Store soon.

The next version of SwiftoDo Desktop will be compiled on the macOS 10.4 SDK, and will no longer support macOS versions lower than that. (If you are not going to upgrade to Mojave, you can continue to use the version of SwiftDo Desktop you are currently using, of course.)

Other than dark mode support, there are no new features. (Sorry!) Because of the under-the-hood changes, and the change in minimum system requirements, I am bumping the version number to 3.0.0, though. (Again, don’t get too excited, but be happy your app is being supported.)

The future of SwiftoDo Desktop

It is a weird coincidence that SwiftoDo and SwiftoDo Desktop will be on the same version number for a while, but it is only a coincidence. At present, they do not share any underlying code.

My long term plan is for SwiftoDo to resemble, and share tons of code with, the iPad version of SwiftoDo. The approach I would prefer to take would be to use the joint iOS/macOS framework that Apple said is coming next year. I will probably continue work on improving the iOS version’s codebase in preparation for eventual macOS support as well.

I am not sure if every user will want the iOS version on the desktop, but I know that I would. I have considered releasing the next-generation version, whenever it is ready, under a new name (SwiftoDo Desktop 2, maybe) and SKU, so owners of the current SwiftoDo Desktop could continue to use it until it no longer functions on macOS. I have not decided exactly what I will do just yet.

I do not expect that anyone outside Apple will learn anything about Apple’s new framework until WWDC 2019, which will be held in June. What I learn at that time will have significant impact on the direction in which I take SwiftoDo Desktop.

Alas, no SwiftoDo update submission to the App Store today

Today I was planning to submit to the App Store the current build of SwiftoDo, which I have been testing without incident for about three weeks. Unfortunately, out of nowhere, it started crashing on launch late this morning. Fortunately, I never finished submitting this build to the App Store for review, so it will never reach users. It does mean, however, that I have to debug a tricky treading-related crash before I can release the cool new feature I have been working on.

SwiftoDo Development Notes, July 2018

I took a break from SwiftoDo development to build a new app, Simple Call Blocker, which I posted about earlier this week. Building it was a fun diversion, and I learned a lot in the process, too. If you’re reading this, you’re probably wondering what’s up with SwiftoDo, however. I have been working on it this summer, too.

Since this spring, I have been promising everyone that I’m working on iOS 11+ Files app integration. This feature will let SwiftoDo open files from any cloud service provider that ties into the iOS 11+ Files app. That includes Dropbox, Box, NextCloud, OneDrive, Google Drive, and many others. Adding additional data providers to SwiftoDo has been a long time coming.

Some background on SwiftoDo data providers

I designed SwiftoDo data providers to be plugins that could be swapped out, which would allow me to support numerous cloud storage providers. I have had so much trouble getting the Dropbox data provider to be solid and stable, however, that I was loath to create more data providers. My thinking was that I was liable to cause more problems for my users (and myself!) than I would solve. Therefore, I concentrated on fixing up the Dropbox data provider code, which is now, after three or four rewrites, pretty solid.

(The Dropbox data provider code has always worked, but it had a rare but nasty crashing bug for some time, and some issues related to stability, ability to handle spotty network connections, and ability to handle, gracefully, Dropbox “rate limit exceeded” API responses.)

iOS 11’s Files app, which I did not anticipate being available when I first wrote SwiftoDo, obviates the need for additional “native” data providers. I just wasn’t sure if SwiftoDo would be able to tie into it.

Can Files integration be used for SwiftoDo?

I have been under the impression that integrating with the Files app would require a rewrite of major sections of the app, or would simply not work well due to sandboxing limitations. I thought this because all Files-based iOS apps that I have used follow Apple’s “Document-based app” template very closely: Think “Microsoft Word” rather than “Reminders” in terms of user interaction patterns. They open to a file browser, you open a document (typically a single document), work on the document, save it (automatically), and close the app. The life cycle of a document is fairly limited: after the app is killed, your file is closed and does not reopen on load. Most of these apps don’t work offline at all, unless you’re working with local-only files, because they can’t open your file.

In contrast, SwiftoDo works a lot differently. It manages your task list locally and syncs it to an external file. It lets you archive completed tasks to a second external file. It keeps the opening and choosing of files down to a minimum. Most users set up their todo.txt and archive files once and never touch those settings again. It lets you work offline (primarily in “manual sync mode”, but also when the network is unavailable) and sync your changes to the cloud on demand.

Integrating SwiftoDo with the Files app, without giving up anything, seemed like it might be a considerable challenge.

A “Files” data provider

To my surprise, implementing a “Files” data provider has not been as challenging as I thought it would be. Accessing documents via the iOS document picker and restoring them after the app is killed, via secure bookmarks, is actually pretty easy. It took me only a couple hours to set up a data provider that would upload and download to iCloud Drive and even Dropbox through the Files app.

That is not nearly enough to ship the feature, though. There are still some issues regarding stability and error handling that I have to work through.

Adding this feature also prompted me to display file names in Settings (rather than 2 screens deep in Settings) and atop the task list in the main view.

What is next?

After I finish the Files data provider work, and the new Xcode and iOS versions are officially released, I plan to build SwiftoDo on the iOS 12 SDK and drop iOS 10 support. I’ve been thinking of bumping the version number to 3.0 at that point, to mark the change in iOS compatibility.

I will also consider the future of Files integration in SwiftoDo. It temping for me to remove the entire “data provider” layer and just make SwiftoDo a normal iOS document-based app. That would be a big deal, and I would not make that change unless I understood fully what that would mean for users. I also have to consider how long I will continue to support the existing Dropbox data provider, as it will be somewhat redundant.

After that, I will have the opportunity to simplify the codebase quite a bit. It is tempting for me to rewrite some or all of the UI layer, to incorporate the new techniques I have learned since coding version 2, over a year ago. Any changes to the UI code will probably be related to new features or a minor redesign of the sorting/filtering interface that I have been thinking about.

Introducing: Simple Call Blocker

Recently I released a new iOS app: Simple Call Blocker.

It is a free utility that lets you block unwanted calls to your iPhone. Unlike most of the call blockers on iOS, it allows you to block whole ranges of numbers, such as your phone number’s extension, for free. You may also whitelist numbers, ranges of numbers, or all your contacts’ phone numbers, so that they will not be blocked by this app, even if they are in the blacklist.

The Simple Call Blocker website explains it in more detail.

Why did I write it?

I wanted to write a small, relatively simple app that would allow me to explore the following things:

  1. iOS application architecture: I have been reading books and articles on iOS application architecture, and decided to create a new app to practice new techniques, such as the use of coordinators for navigation rather than storyboard segues, that I have been learning.
  2. algorithms and operation queues: I started the app by writing some simple algorithms for creating sequential phone number ranges to load into an iOS CallKit directory extension. The last thing I wrote was a multi-threaded operation queue for processing blacklist and whitelist rules and refreshing the iOS call directory extension that actually blocks the phone numbers.
  3. Core Data: I avoided learning Core Data for years; that changed with this app.

Overall, the app was a lot of fun to write. It took me about a month in my after-hours “free time” to create. The overall process has made me a better iOS app developer. I’m excited to bring forward the skills and concepts I developed on this project to future work.

Why a call blocker app?

I started getting neighbor spam calls, so I downloaded nearly all the iOS call blocking apps I could find. There were fewer such apps than I thought there would be, all of them seemed amateurish in some way or another, and all of them (as of a month ago) required an in-app purchase or a subscription to block my neighborhood exchange. I wasn’t willing to pay for that feature in any of the apps that I tried, because all of those apps weren’t that good. Plus, blocking an exchange, or a continuous range of numbers, is pretty trivial, so I thought I could create an app that did that, and offer it for free to people.

So is this just a “practice” app?

No. It is well-written and works as well as iOS’s Call Directory extensions allow it to. It has a rough edge or two, though, in that it reloads its directory extension and reports success or errors back to the users, rather than trying to prevent the user from ever encountering an error from being reported by the directory extension loading process.

What that means is that you can ask Simple Call Blocker to block more numbers that iOS will allow—there is an undocumented limit—and the app has to wait for the CallKit Directory Extension’s load process to report success or failure before it can tell the user what is going on. I decided not to try to impose limits on how many numbers could be blacklisted, but instead allow the app to report the Directory Extension’s errors, if any, back to the user. I figure that the undocumented maximum number of blocked phone numbers probably is dependent on whatever hardware is in your phone, and probably is increasing in every new iPhone model.

The Simple Call Blocker directory extension is coded extremely conservatively, and it optimized for very low RAM and system usage. If iOS cannot load it, it is because the user put too many numbers in the blacklist, or because it is still loading numbers from a prior attempt.

Releasing the app for free, I think, makes it OK that it may not work exactly as users expect it to.

What is the future of this app?

I plan to support it through various iOS releases, but otherwise not improve it too much. After all, it is a free app.

I want to know more!

Go ahead and download the app in the iOS App Store (it’s free!), and check out the FAQ online.

A git commit somehow disappeared, eradicating all my app code and obliterating at least one evening’s worth of work. I cracked open a Guinness, restored a backup from Arq, recovered about 80% of my work, and re-did the rest. I’m happy I didn’t lose my cool.

I’m still testing and fixing things in my dev build of SwiftoDo 2.12.0. It feels weird to have gone so long without issuing a release. I hope to get everything done next week. I have a very full weekend of family activities ahead of me, though, so I can’t do much till Monday.

SwiftoDo Development Notes, April 2018

Weekly updates to SwiftoDo came to an end in early April, but work on SwiftoDo has continued apace.

What’s next?

I am working on an update, version 2.12.0, that includes a couple minor, but long-requested features: (1) a setting to preserve priority on completed tasks and (2) a default priority setting for new tasks. Implementing these features required lots of behind-the-scenes effort. Consequently, neither could be completed in less than a week.

I am currently working on improving the Dropbox code that (1) checks whether SwiftoDo is authorized to access Dropbox, and (2) reports this to the user in a clear and actionable way. This is necessary because Dropbox can de-authorize SwiftoDo for various reasons, including when I upgrade the Dropbox library I am using, which is exactly what the version I am working on does. When this happens, SwiftoDo will alert the user after an upload or a download fails. Based on user reports, however, this notification doesn’t always happen, which can lead to data loss if the user does not realize you are working offline.

Once I finish my work on the Dropbox-related code, I can release this version.

What’s after that?

After I complete version 2.12.0, I plan to focus my efforts on implementing iOS 11 Files integration. Everything else, other than fixing critical bugs, will be put on hold.

The basic mechanics of Files integration are not hard, but they are not really meant for a to-do app—especially one that manages two files. I am unsure if it would require uses to re-open their todo.txt and archive files periodically, after the app is killed, or every time you wish to archive, which may be annoying to users. I am not yet sure how it will affect archiving, manual sync mode, and whether offline access would be possible.

In a best-case scenario, Files integration will eventually allow me to get rid of the Dropbox-related code within SwiftoDo, and rely on Apple’s and Dropbox’s native integration.

In a worst-case scenario, I won’t be able to get Files integration working without giving up too many features or conveniences of the current app.

So, after version 2.12.0 is released, you may not hear from me for some time about development, but I will be hard at work nonetheless.

SwiftoDo Development Notes, March 2018

Today I released the ninth update to SwiftoDo in about ten weeks. What is driving all these small (but good!) releases? Two main things:

  1. I want the app to get better
  2. I want to have fun

I want it to get better

SwiftoDo is a good app, but it is by no means perfect. There are a lot of things that can be improved. My development task list for the app is a mile long. For a long time, the most important items on that list were also the most difficult for me to implement. To be honest, some of those “most important” improvements feel like they are beyond my current capabilities as a developer—but that doesn’t mean that I can’t make improvements somewhere. The app can still get better.

Sometimes, small things can make the app a lot better. Based on many emails with customers, I have learned that, a lot of times, a simple-to-implement feature, rather than a broad reimagining of a portion of the app, will make a big difference to their enjoyment of the app and the productivity they gain from it. That’s why I have been working on “small” features, such as the full file editor, that merely build on what was already there, but end up making the app more powerful and flexible for users. That is also my rationale behind improving application performance, which has become a much higher priority for me this year. Better performance benefits everybody.

I also decided to release features and fixes regularly and frequently. Every week I ask myself, “How can you make the app better for your customers?” And, on another day each week, I ask myself, “Is my latest commit better than what my customers have?” Once I’m sure the new version is better than the last version, I release it.

I figure that adding small features and fixing small bugs eventually accumulates, and my good app can eventually become a great app.

I want to have fun

I’m working on SwiftoDo because it the app is useful to me and because it is fun.

Coding is fun for me, but certainly not every minute of it. Sometimes I have to fight with UIKit’s quirks or work around its bugs, which can take hours of frustrating work. Sometimes I fail to get a feature working without introducing a crash or breaking something else in the app. Sometimes things just don’t work, and it’s really hard to figure out why. Sometimes I’m stuck, and that’s no fun.

I have decided not to remain stuck for more than a day or two anymore. If something isn’t working, I table the work and move onto smaller, solvable problems for a while. This philosophy has led me to work on features that seem simple, useful, and fun to code, but maybe not as important as the larger, more difficult things that have been blocking my progress. That explains why I’ve been pushing forward on improvements to the task text editor, for example, rather than adding new data providers. As a side benefit, working on those smaller things sometimes clears a way, either in the codebase or in my mind, to tackle those larger, more important items.

So, what’s fun? Racking up win after win, week after week, by pushing a better version of my app out to my users. And knowing, every day, that no matter what is not in the app yet, what is in the app keeps getting better.

Version numbers

SwiftoDo’s version number, currently at 2.9.2, is heading into the weird-looking, double-digit-minor-version-number terrority. The next version I release will be 2.10.0.

As Apple suggests, I’m using a 3-number semantic version numbering system, with my own rules for what increments each component. Architectural changes to the app (such as a total rewrite) will bump the first number. Adding new user facing features will bump the second number. Fixing bugs or enhancing existing features, in minor ways, will bump the third number.

Because I am releasing so often now, and batching fewer new user-facing features together, the minor version number has been increasing rapidly. No one should care what the version number is, as long as it goes up. I don’t really care if it is, eventually, version 2.50.0. I does look a little funny to me, though.

What about the Mac version?

I have not been releasing updates to SwiftoDo Desktop recently. The main reason for that is that SwiftoDo Desktop is, basically, feature complete. Unfortunately, because it is coded in Objective C and relies on cell-based table views (mainly for the inline editing to work), it sits at a technological dead end. A total rewrite is in order.

I have prepared for this scenario. My todo.txt-related code is in a framework that can be ported over to the Mac easily. In fact, I have started and stopped a total rewrite of the Mac version a couple times now, but have never gotten that far into it. The things holding me back are:

  1. I have to update my knowledge of AppKit, which is the Mac’s UI framework.
  2. The desktop app uses a different filtering system, which is a little harder to use than the iOS version’s filtering system, but it is much more powerful. I don’t really want to kill it off.
  3. SwiftoDo on iOS could always use more work, and it represents 70% of my user base.

In June, Apple may announce a new framework that would allow me to port my iOS code to the Mac much more easily. If that happens, my ability to provide an updated Mac version would be greatly improved.

Another SwiftoDo update will come at the end of next week, as long as I can get the timing right with App Store review. I cannot promise weekly updates forever, but right now I am trying to release the best version possible to my users, as soon as possible.

SwiftoDo 2.7.2 is now available in the App Store. It adds the ability to drag tasks to new priorities, projects, contexts, tasks, and dates, while in “edit” node, depending on sort order.

SwiftoDo Developer Notes, February 2018

SwiftoDo is a passion project for me. I love working on it, but, due to work and family obligations, I have very little time to do so. Consequently, I am way behind schedule in adopting features introduced in iOS 11. I also learned, the hard way, that lots of minor UI-related bugs popped up when I changed the app’s target iOS framework from 9.0 to 11.0. I have been slowly discovering and cleaning up those bugs, and adding minor features here and there, for the past two and a half months.

Release cadence

I have decided to release working code as soon as possible, rather than trying to batch features and bug fixes into larger releases. Therefore, I have been issuing new releases about once per week, the past few weeks. I will not be keeping up that release cadence, but I do want to reflect to my customers that the app is actively developed. More importantly, I want bugs to be fixed for all my users. I would much rather have a rock-solid, very simple app than an unstable one with lots of bells and whistles.

Features

That said, I do want to keep adding bells and whistles. I am working on adding drag-and-drop support at this point, and plan to look into adding clickable URLs and Siri support. I have a long list of other ideas and concepts drafted, too.

I would like to add additional data providers, other than Dropbox, but I have little exposure to coding networking code, and the third party libraries I’ve looked at look like more trouble than they are worth. For some perspective, Dropbox’s SwiftyDropbox library, which powers file sync now, is a great library, but is also the source of most of the mysterious crashes on startup that a small number of people have reported. What is frustrating to me, as a developer, is that I can’t really fix those crashes, because I don’t fully understand what causes them, and the code is in a library. I don’t want to open my app up to more instability just to add a data provider. Also, I have been loath to support iOS 11’s Files app integration, up to this point, because I don’t see how the todo.txt “Archive” function, which moves completed tasks to another file, would be able to work with it.

Current focus

I have way more ideas for features and improvements than time to complete them. My focus in the near term will be on stability and satisfying user requests that seem like they would be useful for a majority of my users. Hopefully that is good enough for now. It’s amazing how much work my simple, text-based task list app has been!