SQLiteConverter at $14.99 only at the MacApp Store!
Limited time offer!!!!
We have just released SQLiteManager 3.7.0.
It includes the following changes:
- Fixed a bug that occurs when multiple rows was selected in the Manage panel
- Added the ability to search for NULL values in the Manage panel
- SQLite databases can now be saved with a .db extension
- Fixed small issues related to editing fields in inline mode
- Other small fixes and optimizations
We have also already submitted the update to the MacAppStore.
We are proud to announce the first public beta version of the new upcoming server.
This is a beta version so it does not require a valid license and it does not require any developer key, it contains an hard coded expiration date set to March 15th, 2011.
Since this is a public beta we decided to skip the installation process (no installers are availables in this version) so you have to use the command line in order to start the server. MacOS X version is a fat 32bit and 64bit executables, while Linux and Windows versions are 32bit only (64bit support will be available in the next public beta).
This new version of the server has a completely rewritten architecture that is able to handle and huge number concurrent connections. Thanks to the new architecture the server is now much more scalable and flexible. For each supported operating system, we now use a state of the art event API (kqueue on MacOSX, epool on Linux and asynchronous events on Windows). Thanks to this new architecture, the cubesql server is one of the few DBMS server out there that is able of solve the C10K problem (more on http://www.kegel.com/c10k.html).
Why version 4?
We developed the first commercial sqlite based server in 2005 (sqliteserver, our internal version 1).
As you probably know the server has been purchased by REAL Software and the first REAL SQL Server was launched in 2008 (realsqlserver was our internal version 2).
The next version after the REAL SQL Server was the REAL Server and it introduced a lot of powerful new features (realserver was our internal version 3).
In May 2010 SQLabs repurchased the server’s IP property and our team worked hard in order to be able to have a rock solid server with excellent performance, greater scalability, powerful features and all of this with a very intuitive and simple set up process. This new server will be branded as cubesql server and our internal version number is 4.
In the next beta version we’ll include plugins, the new REAL Studio plugin, the new C SDK, the new PHP plugin and the new ODBC drivers for MacOS X, Linux and Windows.
We should also be able to offer 64 bit executables for all the platforms we support and we’ll probably be able to activate the built-in lua scripting language and the built-in webserver.
You can download the public beta from:
A couple of weeks ago we reported an issue to Dr. Hipp about sqlite 3.7.4, the problem we experiencing with the upcoming server was that when there are some concurrent readers (even two or three) over the same table at the same time and the query is a complicated one (a query that required a lot of reads inside the table in order to be able to determine a row that satisfy the SELECT statement) then the time increase exponentially and most of the time the server just hangs.
We analyzed the problem and we discovered that most of the CPU time was spent in System space and not in User space and all the threads seemed blocked inside a semaphore_wait_signal_trap function inside the pcache module.
Dr. Hipp and the sqlite team analyzed the issue and his reply was “As best I can tell, the three threads seem to be colliding in the page-cache module (pcache1.c). The default page-cache is designed for use on low-memory cellphones and gadgets. A common poll of available pages it kept and a mutex is required to manage that pool. That is the design that works best if your goal is to minimize the amount of memory used. But this seems suboptimal on a workstation with lots of memory. On a workstation it seems like giving each connection its own pool so that it does not have to mutex in order to fetch or release a page might be a big win.”
Few hours later I received an experimental code to test and things was a lot better as you can see from the following CPU time graph where most of the time is now spent in User space instead of in System space:
It was just a preliminary test code and yesterday I received the official code that will be part of the upcoming sqlite 3.7.5 with the following advice: “Mutexes are avoided as long as you omit SQLITE_ENABLE_MEMORY_MANAGEMENT. If you do include SQLITE_ENABLE_MEMORY_MANAGEMENT then the mutexes will be engaged again.”
I re-run my tests and I was happily surprised when I watched at the CPU time graph:
Read collisions are now completely avoided and all the bottleneck has been eliminated by design now… queries are now much more fasters and CPU time is all spent in User space. Very good job sqlite team, the upcoming server is now much more faster.
Since the new C SDK will be the base for every server’s clients will write in the future (starting from the REALbasic plugin) I decided to invest some more time trying to optimize the C SDK.
I wasn’t satisfy by the old approach (the base algorithm hasn’t been touched in the latest years) so I decided to completely rewrite the way we download and interpret cursors from the server, trying to reduce as much as possible the reallocation of heap memory and the allocation of index buffers used to refers to the various field inside a cursor.
The old approach seemed fast enough for the average users but it performed very badly for very big cursors. I benchmarked the old way and thew new way using the same exact server, so the speedup you’ll see will be just referred to client side (the new server is already way faster than the old one).
I performed two queries, the first one select everything from a table with 50K rows and 6 columns, the second one select everything from a table with 400K rows and 10 columns (without any index because I just would like to test the raw client speed, I wasn’t interested in the server’s speed).
Here you go the results:
- Test 1: 50K rows and 6 columns
- old client speed: 0.59 secs
- new client speed: 0.51 secs
- speed up about 15%
- Test 2: 400K rows and 10 columns
- old client speed: 41 secs
- new client speed: 7.8 secs
- speed up about 525%
As you can see for very large queries the speedup is quite impressive and this is due just to the way memory is managed in the new SDK. I forgot to mention that since memory is now allocated in a more efficient way, even the memory required to perform large queries is greatly reduced.
We are proud to announce a major upgrade for SQLiteConverter, it now support ODBC databases too!
So it can now natively convert mySQL, postgreSQL and Oracle databases plus a lot of ODBC compliant data sources like Microsoft SQL Server, Access databases and FoxPro (plus many many more).
More information at: