Importing a local storage csv into sqlite database table?

[d_stranz]

I guess what I am wondering is if the steps above, along with the additional code required to insert the data into the database, should be incorporated into the network download .cpp versus in my main.cpp like it is now (separate from the network download code. I pretty much just need some clarification as to how to go about reading the csv files and importing them into the local sqlite tables.

You can put code anywhere you want as long as it works and does what you want it to.

However, if you want to create a robust and maintainable app that can be supported by the next developer to come along after you no longer have responsibility for the code, then I would suggest that you factor your design into components that have as little dependence on each other as possible.

From the description you have provided, you have at three or four independent things going on:

1 - You have an app that needs to be run periodically. This is what main.cpp is for, and nothing else. It creates the QApplication instance that runs the Qt part of your app, creates the top-level UI component, and starts it running. That's it - a typical 5 or 6 line main.cpp like you see in every Qt example app.

2 - You have a component that needs to monitor an ftp site for updates to a few files. When it sees a change, it downloads the files to a known location on disk.

3 - You have a component that is responsible for maintaining an up-to-date SQLite database. It does this by reading in CSV files and updating the database.

Components 2 and 3 are completely unrelated to each other. The only thing they have in common is that they each know about a certain location where they write or read CSV files. Component 2, the network component doesn't need to know anything about the content of the files, just how to download them and put them in the right place. Component 3, the database update component, doesn't need to know anything about where the files came from, just that when new ones show up, it must update the DB.

So at a minimum, you need three source code files: one containing main(), one pair of cpp and h files to implement the class for component 2, and one pair of cpp and h files to implement that class for component 3.

Component 2's only job is to watch the ftp site and download files when there are changes. The is basically the network example, customized. Maybe this class is called FTPMonitor, and it has methods like checkForChanges() and downloadFiles().

Component 3's only job is to wait for new files and update the database when they show up. This includes the code you posted above. Maybe that class is called DBUpdater, and it has methods like checkForNewFiles() and updateDatabase().

You don't say if your app runs continuously like a service, or if it is only run on demand or according to a schedule. If it does not run continuously, then all component 2 needs to do is to compare timestamps for the local files vs. those on the server and download the new ones.

For component 3, it doesn't matter if the program runs continuously or not. It doesn't do anything if there are no changes.

You could look at Qt's QFileSystemWatcher class, which does exactly what the name implies - it watches files and/or directories for changes, and emits signals when they do. When the program starts up, the DBUpdater component starts the file system watcher on the files or directory and then does nothing else. If the FTPMonitor class downloads something, the QFileSystemWatcher class will emit a signal, which the DBUpdater class uses to start the update.

Alternatively, the FTPMonitor class could have noUpdateRequired() and updateRequired() signals. The first one is emitted after checking the server and finding no changes. The second one is emitted when there are changes, but only after the files have been downloaded and are ready to go.

The DBUpdater class has slots that listen for both of these signals. If it receives the first one, it can call quit() on the app. If it receives the second one, then it updates the DB files and then quits.

A lot of words. The basic idea is that the most maintainable apps are those that clearly separate different functions into separate classes and source code files, don't mix together source code parts that have no relationship to each other, and which minimize the connection between different classes and functions.

[luckachi]

@ d_stranz - thank you for your time with breaking things down and explaining everything, I appreciate it. I will definitely read through your response in more detail but I wanted to respond quickly to one of your questions - this app will be running on mobile devices, phones / tablets for iOS and Android, or at least that is the goal. We currently use development software that costs quite a bit of money that we would like to move away from and develop using something a bit more cost effective and reliable.

[d_stranz]

Can't help you there. I have no experience writing for mobile devices, so I don't know what parts of Qt work and which don't. I am sure there are ports of SQLite for mobile OS (it is implemented in a single source code file) and I would be surprised if Qt's network app didn't work.