implementing high performance distributed filesystem/database

I need to implement the fastest possible way to store a key/value pair in a distributed system on Linux. Records of the database are tiny, 256 bytes on average.

I am thinking to use open(), write() and read() system calls and write the key-value pairs directly at some offset in the file. I can omit fdatasync() system call since I will be using SSD disk with battery, so I don't have to worry about ACID compliance if an unexpected shutdown of the system happens. Linux already provides disk cache implementation, so no reads/writes will happen on sectors that were already loaded in memory. This (i think) would be the fastest way to store data, much faster than any other cache capable database engine like for example GT.M or Intersystem's Globals.

However the data is not replicated, and to achieve replication, I can mount a filesystem of another Linux server with NFS and copy the data there, so for example, if I have 2 data servers (1 local and 1 remote), I would issue 2 open(), 2 write() and 2 close() calls. If a transaction fails on remote server, I would mark it as "out of sync" and simply copy the good file again when the remote server comes back.

What do you think of this approach? Will it be fast? I can use NFS over UDP so I will avoid the TCP Stack overhead.

Advantage list so far goes like this:

  • Linux disk cache reused
  • Few lines of code
  • High performance

I will be coding this in C. To locate the record in the file I will keep a btree in memory with a pointer to physical location.

-------------Problems Reply------------

A few suggestions come to mind.

  • is it necessary to open()/write()/close() for every transaction? the system call overhead of open() in particular is probably non-trivial
  • could you use mmap() instead of explicit write()s?
  • if you're doing 2 write() calls (1 local, 1 NFS) for each transaction, it seems like any kind of network problem (latency, dropped packets, etc.) has the potential to bring your application to a screeching halt if you're waiting for the NFS write() call to succeed. And if you're not waiting, for example by doing the NFS writes from a separate thread, your complexity will rapidly grow (I don't think "Few lines of code" will remain true.)

In general, I would suggest that you really prove to yourself that the available tools don't meet your performance requirements before choosing to re-invent this particular wheel.

You might look into a real distributed filesystem rather than using NFS, which as you point out, still provides a single point of failure and no replication.

The Andrew File System (AFS) originally developed by CMU may be a solution for you. It's a commercial product, but you might check out OpenAFS which works on linux (and other systems).

Warning though: AFS has a learning curve.

Category:c# Views:0 Time:2012-01-10
Tags: database linux

Related post

  • How to implement a very simple filesystem? 2011-01-17

    I am wondering how the OS is reading/writing to the hard drive. I would like as an exercise to implement a simple filesystem with no directories that can read and write files. Where do I start? Will C/C++ do the trick or do I have to go with a more l

  • How do I "distribute" a database? 2011-02-15

    Until now my "database server" has been happy with being a single PC. Unfortunately the load is becoming too much, so it looks like I need to "distribute" the database. What exactly does that entail? How can I spread my d/b across several servers? An

  • Checklist towards optimal performance in a database import application 2009-01-12

    I am facing an application designed to import huge amounts of data into a Microsoft SQL Server 2000 database. The application seems to take an awful long time to complete and I suspect the application design is flawed. Someone asked me to dig into th

  • How to increase performance for MySQL database 2009-01-16

    How to increase the performance for mysql database because I have my website hosted in shared server and they have suspended my account because of "too many queries" the stuff asked "index" or "cache" or trim my database I don't know what does "index

  • Does performance of a database (SQL Server 2005) decrease if I shrink it? 2009-07-12

    Does performance of a database (SQL Server 2005) decrease if I shrink it? What exactly happen to the mdf and ldf files when shrink is applied (Internals???) --------------Solutions------------- When shrinking a database it will consume resources to s

  • How to increase the performance of a Database? 2010-01-05

    I have designed databases several times in my company. To increase the performance of the database, I look for Normalisation and Indexing only. If you were asked to increase the performance of a database which has approx 250 tables and some tables wi

  • How can I tie togeather extra space on Macintosh desktops with a distributed filesystem? 2010-09-07

    I have access to a bunch of Mac desktops, the hard drives of which are under-utilized. I want to set up a distributed filesystem to gang them together into one large virtual volume. The server has to be able to run as a normal user. I've tried PVFS2,

  • Performance of SQLite database with too many columns 2011-03-24

    I am developing an iOS application, and the performance is really important for my application. It has a bookmark database that may contain up to 1000 rows, depending on the usage of each user. Right now, each row has 10 columns and most of them are

  • Are there any easy to use distributed relational databases? 2011-07-18

    I've been talking with one of my friends about distributed relational databases, using the defintion: Supports everything a relational database does (SQL, ACID) Distributed (multiple servers) Automatic (or automatic-ish) -- Install the program on mul

  • Is there any implementation of Transparent Distributed Queries over a graph database? 2011-10-11

    I'm designing a P2P application in which every peer have a partial graph. Some nodes in the partial graph can be proxy nodes which contain contact information for another peer. That peer's partial graph includes that proxy node (and potentially other

  • implementing a Distributed System/database 2010-02-19

    We are implementing a system in the company I work for where by we will need to install the system in various sites of the same client (warehouses). The users in all sites should see the same information. The system should be able to work in each sit

  • Looking for distributed/scalable database solution where all nodes are read/write? Not MongoDB? 2011-03-25

    I'm looking to implement a database that can be widely distributed geographically and such that each node can be read/write with eventual consistency to all other nodes. Where should I be looking? I thought that MongoDB seemed like a nice option for

  • Improving query performance of of database table with large number of columns and rows(50 columns, 5mm rows) 2012-01-21

    We are building an caching solution for our user data. The data is currently stored i sybase and is distributed across 5 - 6 tables but query service built on top of it using hibernate and we are getting a very poor performance. In order to load the

  • Implementing functionality/code directly in database system 2008-10-08

    RDBMS packages today offer a tremendous amount of functionality beyond standard data storage and retrieval. SQL Server for example can send emails, expose web service methods, and execute CLR code amongst other capabilities. However, I have always tr

  • char vs varchar for performance in stock database 2008-12-08

    I'm using mySQL to set up a database of stock options. There are about 330,000 rows (each row is 1 option). I'm new to SQL so I'm trying to decide on the field types for things like option symbol (varies from 4 to 5 characters), stock symbol (varies

  • Business Objects XI performing DISTRIBUTED TRANSACTION by default-How to turn off? 2009-07-17

    I have a Business Objects Web Intelligence report that is performing a distributed query against two databases. When I run this query that includes a BEGIN DISTRIBUTED TRANSACTION QUERY SET xact_abort ON GO USE Pipeline2 GO BEGIN DISTRIBUTED TRANSACT

  • DDD - How to implement high-performing repositories for searching 2010-01-19

    I have a question regarding DDD and the repository pattern. Say I have a Customer repository for the Customer aggregate root. The Get & Find methods return the fully populated aggregate, which includes objects like Address, etc. All good. But whe

  • How to Implement Web Based Find File Database Via Text Search 2010-04-20

    I have series of files like this: foo1.txt.gz foo2.txt.gz bar1.txt.gz ..etc.. and a tabular format file that describe those files: foo1 - Explain foo1 foo2 - Explain foo2 bar1 - Explain bar1 ..etc.. What I want to do is to have a website with a simpl

  • Looking for a good implementation that will access multiple databases simultaneously 2010-04-30

    I just want to find out if there's a good implementation (in C#/ASP.NET) on how to access my records from my different databases simultaneously. I have this simple application which suppose to search on these databases, then, display the record if a

Copyright (C), All Rights Reserved.

processed in 0.123 (s). 11 q(s)