Tuesday, February 10, 2015

What Is That Light-Green Oracle Database CPU Wait Time?

This page has been permenately moved. Please CLICK HERE to be redirected.

Thanks, Craig.


What Really Is That Light-Green Oracle Database CPU Wait Time?



Have you ever wondered what that light-green "cpu wait time" really means in Oracle Enterprise Manager? It's what I call, the "gap" time. The "gap" time is the "missing" or the "leftover" time when DB Time does not equal the DB CPU (foreground process CPU consumption) plus the non-idle wait time. And, it happens more often than you might think.

If you have ever noticed that the database time seems too large, then you need to read this article. And, if you really want to know what the light-green "cpu wait time" in your OEM charts is, then you need to read this article. It's that good.

If you're serious about Oracle performance tuning and analysis, you'll want to know I just posted my complete 2015 public training schedule. It's on the main OraPub.com page HERE. Remember, alumni receive a 50% discount...crazy I know.

My Experiment Shows...


My experiment shows a strong relationship between the "gap" time and operating system CPU utilization. This means that a significant portion of the "gap" time is Oracle foreground processes sitting in the CPU run queue ready to consume CPU. This CPU run queue time is not part of DB CPU but it part of DB Time. So, when the CPU run queue time increases, so does DB Time and so does the "gap" time. And I have the data to show it! And you can run the same experiment yourself.

Let me put this another way. Most of the DB Time "gap" is Oracle foreground processes waiting in the operating system CPU run queue so they can eventually and truly consume CPU.

This is really important: When an Oracle foreground process is not consuming CPU but is sitting in the CPU run queue, Oracle Active Session History (ASH) facility records the session sample state as "CPU" and if the Oracle process is a foreground process (not a background process) Oracle's time model records this time as DB Time but not DB CPU. So in both the ASH and time model cases, someone must do some math to calculate this "cpu wait time".

But that name... "cpu wait"!

CPU Wait Time Is A Lousy Name


"CPU wait time" is a lousy name. Why? Mainly because it has caused lots of confusion and speculation. The name would be more appropriately called something like, "cpu queue time." Three reasons come to mind.

First, wait time means something special to Oracle DBAs. To an Oracle DBA anything associate with a "wait" should have a wait event name, a wait occurance, the time should be instrumented (i.e., measured) and should be recorded in the many wait interface related views, such as v$system_event or v$session.

Second, from an Oracle perspective the process is truly "on cpu" because the process is not "waiting." Remember, an Oracle session is either in one of two states; CPU or WAIT. There is no third choice. So the words "CPU Wait" are really confusing.

Third, from an OS perspective or simply a non-Oracle perspective, the Oracle process is sitting in the CPU run queue.

I'm sure in some Oracle Corporation meeting the words "cpu wait" were considered a great idea, but it has caused lots of confusion. And I'm sure it's here to stay.

What Does This "CPU WAIT" Look Like In OEM?


In OEM, the "cpu wait" is a light green color. I grabbed a publically available screenshot off the internet and posted it below. Look familiar? 


OK, so it's really easy to spot in OEM. And if you've seen it before you know EXACTLY what I'm referring to.

What Is CPU Wait Time?


First, let's review what we do know.

1. DB CPU is true Oracle foreground process CPU consumption as reported by the OS through a system call, such as getrusage.

2. CPU Wait time is derived, that is, somebody at Oracle wrote code to calculate the "cpu wait" time.

3. CPU Wait time is a lousy name because it causes lots of confusion.

4. CPU Wait time is shown in OEM as a light green color. DB CPU is shown as a dark/normal green color.

Second, I need to define what I'll call the DB Time "gap." This is not error and I am not implying something is wrong with database time, that it's not useful or anything like that. All I am saying is that sometimes DB Time does not equal DB CPU plus the non-idle wait time. Let's put that in a formula:

DB Time = DB CPU + non Idle Wait Time + gap

Really, What Is CPU Wait Time?


Now I'm ready to answer the question, "What is CPU WAIT time?" Here is the answer stated multiple ways.

"CPU Wait" time is Oracle foreground process OS CPU run queue time.

I ran an experiment (detailed below) and as the OS CPU utilization increased, so did the DB Time "gap" implying that the gap is CPU run queue time or at least a significant part of it.

I ran an experiment and there was a strong correlation between OS CPU utilization and the DB Time "gap" implying that the gap is CPU run queue time.

I ran an experiment and using queuing theory I was able to predict the "gap" time implying that the gap is CPU run queue time. (Whoops... sorry. That's what I'll present in my next post!)

So I'm very comfortable stating that when DB Time is greater than Oracle process CPU consumption plus the non-idle wait time, it's probably the result of Oracle foreground process CPU run queue time.

Yes, there could be some math problems on Oracle's side, there could be uninstrumented time (for sure it's happened before), the operating system could be reporting bogus values or a host of other potential issues. But unless there is an obvious wrong value, I'm sticking with the experimental evidence.

Now I'm going to show the experimental "evidence" that is, that the DB Time "gap" time correlates with the OS CPU utilization.

Let The Data Drive Our Understanding


You can download all the data collection scripts, raw experimental data, Mathematica notepad files, graphic files, etc HERE in a single zip file.

You should be able to run the experiment on any Linux Oracle test system. All you need is a logical IO load and for that I used my free opload tool which, you can download HERE.

The experiment placed an increasing logical IO load on an Linux Oracle 12c system until the operating system CPU utilization exceeded 90%. The load was increased 18 times. During each of the 18 loads, I gathered 31 three minute samples. Each sample contains busy time (v$osstat), idle time (v$osstat), logical IO (v$sysstat "session logical reads"), non-idle wait time (v$system_event where wait_class != 'Idle'), DB CPU (v$sys_time_model), background cpu time (v$sys_time_model), database time (v$sys_time_model DB time) and the sample time (dual table current_timestamp).

The CPU utilization was calculated using the "busy idle" method that I blog about HERE. This method is detailed in my Utilization On Steroids online video seminar.

The workload is defined as the logical IOs per second, lio/s.

Below is a table summarizing the experimental data. The times shown are the averages. If you look at the actual raw experimental data contained in the analysis pack, you'll notice the data is very consistent. This is not suprising since the load I placed should produce a very consistent workload.

Do you see the gaps? Look closely at load 18. The DB Time is 8891.4 seconds. But the sum of DB CPU (996.8 seconds) and the non-idle wait time (2719.2) seconds only equals 3716.0. Yet DB Time is 8891.4. So the "gap" is 5175.3 which is DB Time (8891.3) minus DB CPU (996.8) minus the non-idle wait time (2719.2).

Note: Load 11 and 12 where excluded because of a problem with my data collection. Sorry.

While we can numberically see the DB Time "gap" increase as the CPU utilization increases, check out the graphic in the next section!

The Correlation Between CPU Utilization And DB Time Gap


We can numerically and visually see that as the CPU utilization increases, so does the DB Time "gap." But is there a strong mathematical correlation? To determine this, I used all the experimental samples (except load 11 and 12). Because there was 17 different workloads and with each workload I gathered 31 samples, the correlation comprises of something like 527 samples. Pretty good sample set I'd say.

The correlation coefficient is a strong 0.891. The strongest is 1.0 and the weakest is 0.

Graphically, here is the scatterplot showing the relationship between the CPU utilization and the workload.

Don't expect the DB Time "gap" and OS CPU utilization correlation to be perfect. Remember that DB Time does not include Oracle background process CPU consumption, yet it is obviously part of the OS CPU utilization.

Summary


My experiment indicated the light-green "CPU wait time" is primarily Oracle foreground process operating system CPU run queue time. This is DB Time "gap" time.

My experiment also showed the "gap" time is highly correlated with CPU utilization. Which means, as the CPU utilization increases, so does the "gap" time.

If there are Oracle Database instrumentation bugs or a host of other potential problems, that will also affect the "gap" time.

If you want a more complete and detailed DB Time formula is would be this:

DB Time = DB CPU + Non Idle Wait Time + gap time

In my next post, I'll show you how to calculate the gap time based on queuing theory!

Thanks for reading!

Craig.








163 comments:

  1. Hi Craig -

    "CPU Wait time is a lousy name because it causes lots of confusion."

    Don't really agree here about the lousy name or the source of confusion, so I wouldn't regard it so much as fact than opinion. Yes the name was decided by committee and maybe it could be better, but "gap time" would not be any less confusing and doesn't easily translate into a relationship with "run queue time". I think the real confusion is due to the fact that this value is derived from other values as you describe, and not measured directly like all the other values on the EM Perf Page. The page should describe the calculation somewhere to eliminate the confusion, not look for a better name.

    One reason this was not called "run queue time" is that it is only a lower-bound estimate of run queue time so we did not want to imply that it was how much run queue time was accumulating altogether. It's important to remember (and you don't mention in your post) that when system is CPU-constrained run queue time will distort the wait event timings, perhaps significantly. That is, some (and maybe lots) of run queue time will be "captured" and accounted as wait time. See my slideshare presentation on ASH for details. This distortion can be really severe in the case of frequent low-latency events like latch gets. If you repeat your experiments with a latch-get-and-CPU-heavy workload and measure the run queue time externally, you should see that the "gap time" only accounts for part of the run queue time, the part when session activity was: CPU-run queue-CPU. When session goes CPU-wait-CPU then the run queue time associated with accessing the CPU after the wait is accounted as part of the wait time (and therefore INCLUDED in DB Time)

    JB

    ReplyDelete
  2. Halo Teman, lama saya cari-cari akhirnya ketemu di blog ini.
    Konten yg sangat menarik, Jadi senang berkunjung di sini, Makasih ya sob.
    Salam best Friend :
    Jual Plat Stainless,
    jual Plat Aluminium,
    Lisati Indonesia,
    Lia Property Indonesia

    ReplyDelete
  3. It is really very excellent,i find all articles was amazing.awesome way to get exert tips from everyone,
    not only i like that post all peoples like that post,because of all given information was wonderful and it's very helpful for me.
    dotnet training in Chennai

    ReplyDelete
  4. This blog provides useful information about new techniques and concepts.very impressive lines are given which is very attractive.
    informatica training in chennai

    ReplyDelete
  5. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us.


    VMware Training in Chennai

    ReplyDelete
  6. This is an awesome post.Really very informative and creative contents. These concept is a good way to enhance the knowledge.I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.

    PPC Services in Chennai

    ReplyDelete
  7. I do agree with all the ideas you have presented in your post. They’re really convincing and will certainly work. Still, the posts are very short for newbies. Could you please extend them a little from next time? Thanks for the post..

    Web development company in Chennai

    ReplyDelete
  8. Thank you for sharing such a nice and interesting blog with us. i have seen that all will say the same thing repeatedly. But in your blog, I had a chance to get some useful and unique information. I would like to suggest your blog in my dude circle. please keep on updates. hope it might be much useful for us. keep on updating...
    Software Testing Training

    ReplyDelete
  9. It's interesting that many of the bloggers to helped clarify a few things for me as well as giving.Most of ideas can be nice content.The people to give them a good shake to get your point and across the command .

    digital marketing company in chennai
    seo company in chennai

    ReplyDelete
  10. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.
    Software Testing Training

    ReplyDelete
    Replies
    1. Really i enjoyed very much. And this may helpful for lot of peoples. So you are provided such a nice and great article within this.

      Hr Consultancy in Chennai

      Delete
  11. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your.
    Digital Marketing Company in India

    ReplyDelete
  12. Great site for these post and i am seeing the most of contents have useful for my Carrier.Thanks to such a useful information.Any information are commands like to share him.

    Seo Training in Chennai

    ReplyDelete
  13. Thanks for sharing this information and keep updating us. This is more informatics and it really helped me to know the software Testing.Check the following links about SoftWare Testing .its really helped.


    software Testing Training in Chennai
    testing training in chennai
    Software testing institutes in Chennai

    ReplyDelete
  14. Thanks for sharing the useful information about the topic and good points were stated in the blog for the further information visit the site
    Oracle Fusion Financials Training

    ReplyDelete
  15. I think this is an great blogs. Such a very informative and creative contents. These concept is good for these knowledge.I like it and help me to development very well.Thank you for this brief explanations.

    Oracle Training in Chennai

    ReplyDelete
  16. Being new to the blogging world I feel like there is still so much to learn. Your tips helped to clarify a few things for me as well as giving..
    Android App Development Company
    iOS App Development Company

    ReplyDelete
  17. Really Good blog post.provided a helpful information.keep updating...
    Digital marketing company in Chennai

    ReplyDelete
  18. Good and interesting article . thanks for sharing your post

    Software Testing Training in chennai

    ReplyDelete
  19. Its a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much

    Installment loans
    Payday loans
    Title loans
    Cash Advances

    ReplyDelete
  20. You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...
    Fitness SMS
    Fitness Text
    Salon SMS
    Salon Text
    Investor Relation SMS
    Investor Relation Text

    ReplyDelete
  21. Superb. I really enjoyed very much with this article here.
    Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your.

    Oracle Fusion Procurement Online Training

    ReplyDelete
  22. very very amazing explaintion....many things gather about yourself...yes realy i enjoy it
    SEO Company in India

    ReplyDelete
  23. very very amazing explaintion....many things gather about yourself...yes realy i enjoy it
    seo company in Chennai

    ReplyDelete
  24. you are posting such a new idea is very interesting and give updates.
    msbi training in chennai

    ReplyDelete
  25. your post conveys a good messages and interesting things and give more updates.
    digital marketing company in india

    ReplyDelete

  26. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic.


    Pawn Shop

    Pawn Loans

    Pawn Shops

    Pawn Loan

    Pawn Shop near me

    ReplyDelete
  27. great and nice blog thanks sharing..I just want to say that all the information you have given here is awesome...Thank you very much for this one.
    web design Company
    web development Company
    web design Company in chennai
    web development Company in chennai
    web design Company in India
    web development Company in India

    ReplyDelete
  28. This is excellent information. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
    Mobile Marketing Service
    Mobile Marketing Companies
    Texting API
    Sms API
    sms marketing

    ReplyDelete
  29. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us.
    Dot Net Training Institute in Chennai

    ReplyDelete
  30. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us.

    Regards:
    Nithya,

    Android Training Institute in Chennai

    ReplyDelete
  31. Really it was an awesome article...very interesting to read..You have provided an nice article....Thanks for sharing..
    Android Training in Chennai
    Ios Training in Chennai

    ReplyDelete
  32. I would like to say that this blog really convinced me, you give me best information! Thanks, very good post.

    graphics design training in indore
    Keep Posting:)

    ReplyDelete
  33. This comment has been removed by the author.

    ReplyDelete
  34. Thanks for sharing the information. It is very useful for my future. keep sharing.

    Best bca college in noida
    Top bca colleges in noida

    ReplyDelete
  35. Awe! What An Blog Very Helpful and interesting Really A great center for acquiring knowledge.Very Helpful Post And Explained Very Clearly About All the things.Very Helpful. Coming To Our Self We Provide Restaurant Equipment Parts .Really Thankfull For the blogger providing such a great information.Thank you. Have a Nice Day.

    ReplyDelete
  36. Whatever We Are Gathering the information from the blogs,we have to keep on updating it,Urgent Care in Chicago.By getting them into one place.Really thanks for posting.Very Thankful for the Informative Post.Really Thanks For Posting.

    ReplyDelete
  37. It's A Great Pleasure reading your Article. USMLE in Chicago Thanks For Posting.

    ReplyDelete
  38. Really Thanks For Posting Such an Informative Article....

    Vizag Real Estate

    ReplyDelete
  39. Our digital marketing course in Chennai is targeted at those who are desirous of taking advantage of career opportunities in digital marketing. Marketing professionals who are presently using traditional marketing to meet their business objectives. Start-ups, Entrepreneurs, Business Owners who desire to make use of online media to improve their business goals and objectives.

    Digital Marketing Course in Chennai
    Digital Marketing Training in Chennai
    Online Digital Marketing Training
    SEO Training in Chennai
    Digital Marketing Course
    Digital Marketing Training
    Digital Marketing Courses

    ReplyDelete
  40. This comment has been removed by the author.

    ReplyDelete
  41. This comment has been removed by the author.

    ReplyDelete
  42. Nice post. By reading your blog, i get inspired and this provides some useful information. Thank you for posting this exclusive post for our vision.
    Selenium Training in Chennai | Selenium Training in Bangalore | Selenium Training in Pune | Selenium online Training

    ReplyDelete
  43. I wish to indicate because of you only to bail me out of this specific trouble. As a consequence of checking through the net and meeting systems that were not beneficial, I thought my life was finished.industrial safety course in chennai

    ReplyDelete
  44. Thanks a lot very much for the high quality and results-oriented help. I won’t think twice to endorse your blog post to anybody who wants and needs support about this area
    nebosh course in chennai

    ReplyDelete
  45. Positive site..Really nice to have read..Thanks for sharing..Keep update php training in chennai
    php training in velacher chennai

    ReplyDelete
  46. Thanks for your great and helpful presentation I like your good service.I always appreciate your post.That is very interesting I love reading and I am always searching for informative information like this.Well written article Thank You for Sharing with Us pmp training fee | pmp certification course in chennai | best pmp training institute in chennai| | pmp training class in chennai \ pmp training fee

    ReplyDelete
  47. Hello! This is my first visit to your blog! We are a team of volunteers and starting a new initiative in a community in the same niche. Your blog provided us useful information to work on. You have done an outstanding job.

    Best AWS Training in Chennai | Amazon Web Services Training in Chennai


    AWS Training in Bangalore | Amazon Web Services Training in Bangalore


    Amazon Web Services Training in OMR , Chennai | Best AWS Training in OMR,Chennai

    ReplyDelete
  48. Thanks for such a great article here. I was searching for something like this for quite a long time and at last, I’ve found it on your blog. It was definitely interesting for me to read about their market situation nowadays pmp training centers in chennai| pmp training in velachery | pmp training near me | project management training in chennai | project management certification online

    ReplyDelete
  49. Thank you for sharing your article. Great efforts put it to find the list of articles which is very useful to know, Definitely will share the same to other forums.

    best openstack training in chennai | openstack course fees in chennai | openstack certification in chennai | redhat openstack training in chennai

    ReplyDelete
  50. Ah,so beautiful and wonderful post!An opportunity to read a fantastic and imaginary blogs.It gives me lots of pleasure and interest.Thanks for sharing.
    Find the Interior Designers in Vijayawada

    ReplyDelete
  51. Laminated Doors manufacturer in hubli
    Thanks for giving great kind of information. So useful and practical for me. Thanks for your excellent blog, nice work keep it up thanks for sharing the knowledge.

    ReplyDelete
  52. I really like your wonderful post. I loved to read such kind of article. and the first time I visit our website. and I happy to be here, thanks for sharing this amazing post
    Morgan Digital Marketing

    ReplyDelete
  53. Thanks for your great and helpful presentation I like your good service. I always appreciate your post. That is very interesting I love reading and I am always searching for informative information like this.AngularJS Training in Chennai | Best AngularJS Training Institute in Chennai|

    ReplyDelete
  54. Hello hope you doing well I want to say that your blog is just amazing. Excellent list and genuine information and it works for me. Thank you for sharing I am looking forward to the next blog.
    Digital Marketing Institute in Delhi
    QuickBooks Error 3371

    ReplyDelete
  55. Outstanding blog thanks for sharing such wonderful blog with us ,after long time came across such knowlegeble blog. keep sharing such informative blog with us.

    machine learning classroom training in chennai
    machine learning certification in chennai
    top institutes for machine learning in chennai
    Android training in velachery
    PMP training in chennai

    ReplyDelete
  56. Such a Great Article!! I learned something new from your blog. Amazing stuff. I would like to follow your blog frequently. Keep Rocking!!
    Blue Prism training in chennai | Best Blue Prism Training Institute in Chennai

    ReplyDelete
  57. Wow!! Really a nice Article. Thank you so much for your efforts. Definitely, it will be helpful for others. I would like to follow your blog. Share more like this. Thanks Again.
    iot training in Chennai | Best iot Training Institute in Chennai

    ReplyDelete
  58. Watch Here Online Pinoy Reply Shows Hd watch latest Pinoy Shows as well as GMA Network Tv episode of Pinoy lambingan
    Ika-5 Utos Pinoy

    ReplyDelete
  59. Very good article thank you for sharing this useful information You casn See Here all Piony Tv Shows
    Halik Pinoy Replay

    ReplyDelete
  60. Thanks for sharing this blog, this blog is very helpful information for every one. You can Watch here Colors Tv Star plus and Zee Tv Episodes
    Watch Internet Wala Love Online

    ReplyDelete
  61. Watch Here Online Pinoy Reply Shows Hd watch latest Pinoy Shows as well as GMA Network Tv episode of Pinoy lambingan
    Ang Probinsyano Pinoy

    ReplyDelete
  62. This blog was really amazing, thank you so much for sharing this blog.
    Lifestyle Magazine India

    ReplyDelete
  63. Excellent blog I visit this blog it's really awesome. The important thing is that in this blog content written clearly and understandable. The content of information is very informative. Thanks for the excellent and great idea. keep blogging
    Oracle Fusion Financials Online Training

    ReplyDelete
  64. Thanks for sharing a useful information.. we have learnt so much information from your blog..... keep sharing
    Oracle Fusion HCM Online Training

    ReplyDelete
  65. Such a nice blog, I really like what you write in this blog, I also have some relevant Information about if you want more information.
    Oracle Fusion SCM Online Training

    ReplyDelete
  66. Excellent blog I visit this blog it's really awesome. The important thing is that in this blog content written clearly and understandable. The content of information is very informative. Thanks for the excellent and great idea. keep blogging
    Oracle Fusion Technical Online Training

    ReplyDelete

  67. Welcome to AZLyrics! It's a place where all searches end! We have a large, legal, every day growing universe of lyrics where stars of all genres and ages shine.

    ReplyDelete
  68. Data Science Certification Courses in Hyderabad. ExcelR is the Best Data Science Training Institute in Hyderabad with Placement assistance and offers a blended.
    https://www.excelr.com/data-science-course-training-in-hyderabad/

    ReplyDelete
  69. Your new valuable key points imply much a person like me and extremely more to my office workers. With thanks; from every one of us.
    apple service center chennai | iphone repair chennai | apple iphone service center in chennai | iphone service chennai

    ReplyDelete
  70. Nice blog..! I really loved reading through this article. Thanks for sharing such a
    amazing post with us and keep blogging... iot training in chennai | iot training in chennai quora | iot training and placement in chennai | iot training center in chennai | best iot training centre in chennai

    ReplyDelete
  71. Valuable information provided, thanks for sharing.
    Have got to know about the Stock Market Course in hyderabad from a friend. Interested may checkout the below link- http://avighnastockmarkettraining.com/

    ReplyDelete
  72. Hey Palak! yes, absolutely i have suggested Stock Market Course to other as i see growing career in near future

    ReplyDelete
  73. Thanks for sharing the valuable information with us. will surely recommend this blog to others digital marketing institute

    ReplyDelete
  74. Thanks for sharing this wonderful content with us hope to see more values from you. Also watch this interesting blog Top 10 places to visit in Shimla

    ReplyDelete
  75. Nice blog..! I really loved reading through this article. Thanks for sharing such a amazing post with us and keep blogging... best angularjs training institute in chennai | angularjs training in omr | angular 4 training in chennai | angularjs training in omr

    ReplyDelete
  76. Visit for Website Designing & Development Company at Ogen Infosystem.
    Website Designing Company in India

    ReplyDelete
  77. بسم الله الرحمن الرحيم عميلنا العزيز نحن نقدم افضل خدمات تنظيف المجالس والشقق بافضل انواع التنظيفات المشهود لها عالميا
    شركة تنظيف مجالس بالطائف
    شركة تنظيف مجالس بجازان
    شركة تنظيف مجالس بحائل
    ونحن فى خماتكم 24 ساعة على مدار الشهر

    ReplyDelete
  78. Awesome article! It is in detail and well formatted that i enjoyed reading. which inturn helped me to get new information from your blog.
    Find the Interior Designers in Vizag

    ReplyDelete
  79. Great post. I was once checking constantly this weblog and I'm impressed! Extremely useful information specially the closing part. I maintain such information much. I was once seeking this specific information for a very long time. Many thanks and best of luck.
    apple mobile service centre in chennai
    apple service center near me

    ReplyDelete
  80. This is the exact information I am been searching for, Thanks for sharing the required infos with the clear update and required points. To appreciate this I like to share some useful information regarding Microsoft Azure which is latest and newest,

    Regards,
    Ramya

    Azure Training in Chennai
    Azure Training Center in Chennai
    Best Azure Training in Chennai
    Azure Devops Training in Chenna
    Azure Training Institute in Chennai
    Azure Training in Chennai OMR
    Azure Training in Chennai Velachery
    Azure Online Training

    ReplyDelete
  81. Nice post. Thanks for sharing! I want people to know just how good this information is in your article. It’s interesting content and Great work.
    Thanks & Regards,
    VRIT Professionals,
    No.1 Leading Web Designing Training Institute In Chennai.

    And also those who are looking for
    Web Designing Training Institute in Chennai
    SEO Training Institute in Chennai
    Photoshop Training Institute in Chennai
    PHP & Mysql Training Institute in Chennai
    Android Training Institute in Chennai

    ReplyDelete
  82. Nice blog..! I really loved reading through this article. Thanks for sharing such a amazing post with us and keep blogging... best angularjs training institute in chennai | angularjs training in omr |

    ReplyDelete
  83. Thanks for sharing such a great blog... I am impressed with you taking time to post a nice info.
    App Development Company

    ReplyDelete
  84. I’m going to read this. I’ll be sure to come back. thanks for sharing. and also This article gives the light in which we can observe the reality. this is very nice one and gives indepth information.
    R Training Institute in Chennai | R Programming Training in Chennai

    ReplyDelete
  85. This is the exact information I am been searching for, Thanks for sharing the required infos with the clear update and required points. To appreciate this I like to share some useful information regarding Microsoft Azure which is latest and newest,

    Regards,
    Ramya

    Azure Training in Chennai
    Azure Training Center in Chennai
    Best Azure Training in Chennai
    Azure Devops Training in Chenna
    Azure Training Institute in Chennai
    Azure Training in Chennai OMR
    Azure Training in Chennai Velachery
    Azure Online Training
    Azure Training in Chennai Credo Systemz
    DevOps Training in Chennai Credo Systemz

    ReplyDelete

  86. It’s amazing in support of me to truly have a web site that is valuable meant for my knowledge.

    Best Ice Fishing Gloves Best Ice Fishing Gloves Best Ice Fishing Gloves

    ReplyDelete
  87. Truly a very good article on how to handle the future technology. After reading your post,thanks for taking the time to discuss this, I feel happy about and I love learning more about this topic.
    - Mumbai Web, Crocus Web

    ReplyDelete
  88. Great stuff you are sharing!
    love your writing skills.
    keep posting things like this.

    Hindi Panda

    ReplyDelete
  89. I feel happy about and learning more about this topic. keep sharing your information regularly for my future reference. This content creates new hope and inspiration within me. Thanks for sharing an article like this. the information which you have provided is better than another blog.
    IELTS Coaching in Dwarka
    Best IELTS Coaching in Dwarka

    ReplyDelete
  90. Hello,
    Nice article… very useful
    thanks for sharing the information.
    servicenow orchestration training

    ReplyDelete
  91. Well article, interesting to read…
    Thanks for sharing the useful information
    apache training

    ReplyDelete
  92. A bewildering web journal I visit this blog, it's unfathomably heavenly. Oddly, in this present blog's substance made purpose of actuality and reasonable. The substance of data is informative
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  93. uvoffer- if you are searching for free unlimted tricks then visit now on Uvoffer.com and get unlimited offers and informations.
    film ka naam whatsapp puzzle answer film ka naam whatsapp puzzle

    ReplyDelete
  94. Whatscr - many peoples want to join random whatsapp groups . as per your demand we are ready to serve you whatsapp group links . On this website you can join unlimited groups . click and get unlimited whatsapp group links

    ReplyDelete
  95. My manju - if you want girls mobile numbers then this website is best for you . you can visit on this website and get their information and you also can meet with thrm and go for a date . click here to use our website --- online dating website

    ReplyDelete
  96. Thank you so much for given such an informative blog. Get the best Website Designing and Development Services by Ogeninfo.
    Website Designing Company in Delhi

    ReplyDelete
  97. Get the best performing Mutual Fund by Mutual Fund Wala and know the best investment schemes.
    Mutual Fund Agent

    ReplyDelete
  98. Thank you for providing such an awesome article and it is a very useful blog for others to read.

    Oracle ICS Online Training

    ReplyDelete
  99. Thanks for providing a useful article containing valuable information. start learning the best online software courses.

    Workday Online Training

    ReplyDelete
  100. Great efforts put it to find the list of articles which is very useful to know, Definitely will share the
    same to other forums.
    big data hadoop training cost in chennai | hadoop training in Chennai | best bigdata hadoop training in chennai | best hadoop certification in Chennai

    ReplyDelete
  101. Thanks for sharing valuable information.It will help everyone.keep Post.
    Kerala Lottery guessing

    ReplyDelete
  102. Thanks for sharing valuable information.It will help everyone.keep Post.
    Dhankesari

    ReplyDelete
  103. Awesome blog again thanks for such an informative blog sharing with us.
    Lifestyle Magazine India

    ReplyDelete
  104. Go Health Science is the best resource to get all kinds of Knowledge about Health and Science updates on Healthy Life ideas.

    ReplyDelete

  105. نجار بجدة
    تعرف مهنة النجارة بأنها واحدة من أهم المهن الذي يترتب عليها أشياء كثيرة، فالنجار هو من يقوم بصناعة الأثاث بمختلف أنواعه وأشكاله، وهو من يقوم بصناعة الأبواب والنوافذ الخشبية، أيضاً نجار بجدة هو الذي يعمل على فك وتركيب غرف نوم بالمدينة المنورة ويقوم بتركيب دواليب، ومن ضمن المهام الذي يشترك بها نجار بالمدينة المنورة هي عملية نقل الأثاث من مكان إلى آخر، فهو من يعتمد عليه في أعمال فك وتركيب الأثاث وأثاث ايكيا بالمدينة المنورة لأنه يمتلك قدرات وإمكانيات كبيرة تساعده للقيام بذلك وأكثر، يحرص معلم نجار بالمدينة المنورة على استخدام الأدوات اليدوية والأدوات التي تستخدم بواسطة الكهرباء وجميعها مهمة بالنسبة له وتساعد في تنفيذ كافة أعماله.

    ReplyDelete
  106. An astounding web diary I visit this blog, it's inconceivably magnificent. Strangely, in this current blog's substance made point of fact and sensible. The substance of information is instructive.
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete