Performance impact of having a data access layer/service layer?

I need to design a system which has these basic components:

  • A Webserver which will be getting ~100 requests/sec. The webserver only needs to dump data into raw data repository.
  • Raw data repository which has a single table which gets 100 rows/s from the webserver.
  • A raw data processing unit (Simple processing, not much. Removing invalid raw data, inserting missing components into damaged raw data etc.)
  • Processed data repository

Does it make sense in such a system to have a service layer on which all components would be built? All inter-component interaction will go through the service layers. While this would make the system easily upgradeable and maintainable, would it not also have a significant performance impact since I have so much traffic to handle?

-------------Problems Reply------------

What do you see as the costs of having a separate service layer?

How do those costs compare with the costs you must incur? In your case that seems to be at least

  1. a network read for the request
  2. a database write for raw data
  3. a database read of raw data
  4. a database write of processed data

Plus some data munging.

What sort of services do you have a mind? Perhaps

  • saveRawData()
  • getNextRawData()
  • writeProcessedData()

why is the overhead any more than a procedure call? Service does not need to imply "separate process" or "web service marshalling".

I contend that structure is always of value, separation of concerns in your application really matters. In comparison with database activities a few procedure calls will rarely cost much.

In passing: the persisting of Raw data might best be done to a queuing system. You can then get some natural scaling by having many queue readers on separate machines if you need them. In effect the queueing system is naturally introducing some service-like concepts.

Here's what can happen unless you guard against it.

In the communication between layers, some format is chosen, like XML. Then you build it and run it and find out the performance is not satisfactory.

Then you mess around with profilers which leave you guessing what the problem is.

When I worked on a problem like this, I used the stackshot technique and quickly found the problem. You would have thought it was I/O. NOT. It was that converting data to XML, and parsing XML to recover data structure, was taking roughly 80% of the time. It wasn't too hard to find a better way to do that. Result - a 5x speedup.

Personally feel that you might be focusing too much on low level implementation details when designing the system. Before looking at how to lay out the components, assemblies or services you should be thinking of how to architect the system.

You could start with the following high level statements from which to build your system architecture around:

  1. Confirm the technical skill set of the development team and the operations/support team.
  2. Agree on an initial finite list of systems that will integrate to your service, the protocols they support and some SLAs.
  3. Decide on the messaging strategy.
  4. Understand how you will deploy your service/system.
  5. Decide on the choice of middleware (ESBs, Message Brokers, etc), databases (SQL, Oracle, Memcache, DB2, etc) and 3rd party frameworks/tools.
  6. Decide on your caching and data latency strategy.
  7. Break your application into the various areas of business responsibility - This will allow you to split up the work and allow easier communication of milestones during development/testing and implementation.
  8. Design each component as required to meet the areas of responsibility. The areas of responsibility should automatically lead you to decide on how to design component, assembly or service.

Obviously not all of the above will match your specific case but I would suggest that they should at least be given some thought.

Good luck.

Abstraction and tiering will introduce latency, but the real question is, what are you GAINING to make the cost(s) worthwhile? Loose coupling, governance, scalability, maintainability are worth real $.

Even the best-designed layered app will exhibit more latency than an app talking directly to a DB. Users who know the original system will feel the difference. They may not like it, so this can be a political issue as much as a technical one.

Category:performance Views:0 Time:2010-08-06

Related post

  • Using a WCF Data Access & Business Logic Layer 2011-08-21

    We are starting development of an web application which would later be ported to mobile as well. So I was thinking to encapsulate Data Access and Business Logic inside an WCF Layer. This would make it easier to sync and manage it later. The data acce

  • Master data validation in service layer 2010-06-17

    User enters the data in the forms by choosing values from the master data drop downs in web layer. Data is populated in the dropdowns based on some logic from the master data tables and we know for sure that its a valid master data as far as web laye

  • How do I (or should I?) access the service layer from a SiteMesh template (views/layouts/main.gsp) in Grails? 2011-01-15

    I need to create a toplist in the page footer on a site that I'm building. The footer is created in the default SiteMesh layout template (views/layouts/main.gsp). In order to create the toplist access to the database is needed, so I've encapsulated a

  • Is there a Performance impact if stored varchar data is less than the defined size limit? 2011-12-06

    If a column defined with varchar(20000) and size of the actual data stored is around 4k-5k, Does it make any difference to MYSQL? As I read from MYSQL documentation, there is no difference in disk usage when actual data is less than declared size. My

  • Should an entity access the Service Layer and execute Update/Save methods? 2011-02-05

    My application requirements: if any property is changed by the user - like entering data into a textbox and then leaving the textbox - it must be updated immediately to the database. I using WPF with MVVM design pattern. All my entities implement INo

  • Should Validation and Data Correction logic be done in business or data access layer? 2011-01-11

    So I have this general question of where should I have certain logic for example - var data=GetItems(); if(data==null) //return some defaults else return values //second case.. if(id<=0) //throw some exception else return id So should the above co

  • Data clean-up; what layer? 2009-11-17

    I have an app built on Model-Glue: Unity that contains some search forms. I need to trim leading and trailing spaces from search strings before using them to query the database. I'm also keeping the search terms in a bean that a user can save and re-

  • Multi-tenant Access Control: Repository or Service layer? 2010-04-30

    In a multi-tenant ASP.NET MVC application based on Rob Conery's MVC Storefront, should I be filtering the tenant's data in the repository or the service layer? 1. Filter tenant's data in the repository: public interface IJobRepository { IQueryable

  • Data Access Object or Inline Methods 2011-01-28

    I am still very new to Actionscript and have an okay working knowledge of Flex. I am working on an AIR application that runs a SQLite database underneath. The database is only 3 tables (projects, items and types). I was wondering if Actionscript best

  • Should a service layer return view models for an MVC application? 2010-06-08

    Say you have an ASP.NET MVC project and are using a service layer, such as in this contact manager tutorial on the site: If you have viewmodels for your view

  • Service layer in java swing application 2010-11-24

    i'm thinking if i really need a service layer. I'm using spring + hibernate for a desktop swing application and at this moment i have gui/swing layer->service layer->dao layer. I use spring only for @Transactional support and for IOC-injection

  • MVC3 Design - repository pattern and services layer 2011-04-13

    I have read a couple of books and articles on MVC, and come across the repository pattern and the services layer. Should a controller be able to get entities through the repository pattern, or must it retrieve data from the services layer? Edit: I ha

  • Should the Domain ever access Application Services of another system? 2011-07-01

    Spefically across two systems: can the domain of system A call the application services/remote facade of system B? For example if an Ordering System has an Order entity in its domain, should the validation method of that Order entity call an applicai

  • Azure, Sync Framework and Access Control Service: Are there obvious shortcomings or problems in using this technologies together? 2011-07-20

    I have a desktop application which uses flat files (some xml and small pictures) as data. I want this data to be available on other PCs which have the desktop application installed and usable by a smartphone client (WP7 at the moment) as well. The us

  • Guidelines for implementing service layer methods 2011-11-16

    We are developing an 3-tier application where presentation layer communicates with service layer, which in turn communicates with the business layer. The business layer has access to database. Currently we have implemented business layer to hold CRUD

  • How to force adding and removing of objects from the entity framework through the service layer? 2011-11-28

    I am struggling with solving how to handle insertions and deletions with MVC3 and the Entity Framework 4.2 My Setup: Website accesses my Service Layer, which uses the Repository Pattern to fetch and return my Model(POCO) objects. The Solution is sepa

  • strategy for maintaining both native and wcf service layer 2011-12-08

    I've got a web application that will sometimes call a service layer from a codebehind as well as call some restful WCF services from clientside code (e.g. JQuery). I've found that there is a bit of work maintaining both service layers. I've considere

  • Service Layer Validation 2011-12-13

    I am trying to implement a validation strategy in my application. I have an MVC layer, Service layer, Repository and Domain POCOs. Now at the MVC layer, i use data annotations on my viewmodels to validate user input, allowing me to give the user quic

  • Should Service layer throw an exception? 2012-05-01

    I don't like throwing exceptions for some reason, maybe because of the performance hit I don't know, wondering if I should re-think this issue. Should my service layer (uses Dao's + business logic etc.) be throwing exceptions? public ModelAndView cre

Copyright (C), All Rights Reserved.

processed in 0.114 (s). 11 q(s)