Tuesday, February 10, 2015

What Is That Light-Green Oracle Database CPU Wait Time?

This page has been permenately moved. Please CLICK HERE to be redirected.

Thanks, Craig.


What Really Is That Light-Green Oracle Database CPU Wait Time?



Have you ever wondered what that light-green "cpu wait time" really means in Oracle Enterprise Manager? It's what I call, the "gap" time. The "gap" time is the "missing" or the "leftover" time when DB Time does not equal the DB CPU (foreground process CPU consumption) plus the non-idle wait time. And, it happens more often than you might think.

If you have ever noticed that the database time seems too large, then you need to read this article. And, if you really want to know what the light-green "cpu wait time" in your OEM charts is, then you need to read this article. It's that good.

If you're serious about Oracle performance tuning and analysis, you'll want to know I just posted my complete 2015 public training schedule. It's on the main OraPub.com page HERE. Remember, alumni receive a 50% discount...crazy I know.

My Experiment Shows...


My experiment shows a strong relationship between the "gap" time and operating system CPU utilization. This means that a significant portion of the "gap" time is Oracle foreground processes sitting in the CPU run queue ready to consume CPU. This CPU run queue time is not part of DB CPU but it part of DB Time. So, when the CPU run queue time increases, so does DB Time and so does the "gap" time. And I have the data to show it! And you can run the same experiment yourself.

Let me put this another way. Most of the DB Time "gap" is Oracle foreground processes waiting in the operating system CPU run queue so they can eventually and truly consume CPU.

This is really important: When an Oracle foreground process is not consuming CPU but is sitting in the CPU run queue, Oracle Active Session History (ASH) facility records the session sample state as "CPU" and if the Oracle process is a foreground process (not a background process) Oracle's time model records this time as DB Time but not DB CPU. So in both the ASH and time model cases, someone must do some math to calculate this "cpu wait time".

But that name... "cpu wait"!

CPU Wait Time Is A Lousy Name


"CPU wait time" is a lousy name. Why? Mainly because it has caused lots of confusion and speculation. The name would be more appropriately called something like, "cpu queue time." Three reasons come to mind.

First, wait time means something special to Oracle DBAs. To an Oracle DBA anything associate with a "wait" should have a wait event name, a wait occurance, the time should be instrumented (i.e., measured) and should be recorded in the many wait interface related views, such as v$system_event or v$session.

Second, from an Oracle perspective the process is truly "on cpu" because the process is not "waiting." Remember, an Oracle session is either in one of two states; CPU or WAIT. There is no third choice. So the words "CPU Wait" are really confusing.

Third, from an OS perspective or simply a non-Oracle perspective, the Oracle process is sitting in the CPU run queue.

I'm sure in some Oracle Corporation meeting the words "cpu wait" were considered a great idea, but it has caused lots of confusion. And I'm sure it's here to stay.

What Does This "CPU WAIT" Look Like In OEM?


In OEM, the "cpu wait" is a light green color. I grabbed a publically available screenshot off the internet and posted it below. Look familiar? 


OK, so it's really easy to spot in OEM. And if you've seen it before you know EXACTLY what I'm referring to.

What Is CPU Wait Time?


First, let's review what we do know.

1. DB CPU is true Oracle foreground process CPU consumption as reported by the OS through a system call, such as getrusage.

2. CPU Wait time is derived, that is, somebody at Oracle wrote code to calculate the "cpu wait" time.

3. CPU Wait time is a lousy name because it causes lots of confusion.

4. CPU Wait time is shown in OEM as a light green color. DB CPU is shown as a dark/normal green color.

Second, I need to define what I'll call the DB Time "gap." This is not error and I am not implying something is wrong with database time, that it's not useful or anything like that. All I am saying is that sometimes DB Time does not equal DB CPU plus the non-idle wait time. Let's put that in a formula:

DB Time = DB CPU + non Idle Wait Time + gap

Really, What Is CPU Wait Time?


Now I'm ready to answer the question, "What is CPU WAIT time?" Here is the answer stated multiple ways.

"CPU Wait" time is Oracle foreground process OS CPU run queue time.

I ran an experiment (detailed below) and as the OS CPU utilization increased, so did the DB Time "gap" implying that the gap is CPU run queue time or at least a significant part of it.

I ran an experiment and there was a strong correlation between OS CPU utilization and the DB Time "gap" implying that the gap is CPU run queue time.

I ran an experiment and using queuing theory I was able to predict the "gap" time implying that the gap is CPU run queue time. (Whoops... sorry. That's what I'll present in my next post!)

So I'm very comfortable stating that when DB Time is greater than Oracle process CPU consumption plus the non-idle wait time, it's probably the result of Oracle foreground process CPU run queue time.

Yes, there could be some math problems on Oracle's side, there could be uninstrumented time (for sure it's happened before), the operating system could be reporting bogus values or a host of other potential issues. But unless there is an obvious wrong value, I'm sticking with the experimental evidence.

Now I'm going to show the experimental "evidence" that is, that the DB Time "gap" time correlates with the OS CPU utilization.

Let The Data Drive Our Understanding


You can download all the data collection scripts, raw experimental data, Mathematica notepad files, graphic files, etc HERE in a single zip file.

You should be able to run the experiment on any Linux Oracle test system. All you need is a logical IO load and for that I used my free opload tool which, you can download HERE.

The experiment placed an increasing logical IO load on an Linux Oracle 12c system until the operating system CPU utilization exceeded 90%. The load was increased 18 times. During each of the 18 loads, I gathered 31 three minute samples. Each sample contains busy time (v$osstat), idle time (v$osstat), logical IO (v$sysstat "session logical reads"), non-idle wait time (v$system_event where wait_class != 'Idle'), DB CPU (v$sys_time_model), background cpu time (v$sys_time_model), database time (v$sys_time_model DB time) and the sample time (dual table current_timestamp).

The CPU utilization was calculated using the "busy idle" method that I blog about HERE. This method is detailed in my Utilization On Steroids online video seminar.

The workload is defined as the logical IOs per second, lio/s.

Below is a table summarizing the experimental data. The times shown are the averages. If you look at the actual raw experimental data contained in the analysis pack, you'll notice the data is very consistent. This is not suprising since the load I placed should produce a very consistent workload.

Do you see the gaps? Look closely at load 18. The DB Time is 8891.4 seconds. But the sum of DB CPU (996.8 seconds) and the non-idle wait time (2719.2) seconds only equals 3716.0. Yet DB Time is 8891.4. So the "gap" is 5175.3 which is DB Time (8891.3) minus DB CPU (996.8) minus the non-idle wait time (2719.2).

Note: Load 11 and 12 where excluded because of a problem with my data collection. Sorry.

While we can numberically see the DB Time "gap" increase as the CPU utilization increases, check out the graphic in the next section!

The Correlation Between CPU Utilization And DB Time Gap


We can numerically and visually see that as the CPU utilization increases, so does the DB Time "gap." But is there a strong mathematical correlation? To determine this, I used all the experimental samples (except load 11 and 12). Because there was 17 different workloads and with each workload I gathered 31 samples, the correlation comprises of something like 527 samples. Pretty good sample set I'd say.

The correlation coefficient is a strong 0.891. The strongest is 1.0 and the weakest is 0.

Graphically, here is the scatterplot showing the relationship between the CPU utilization and the workload.

Don't expect the DB Time "gap" and OS CPU utilization correlation to be perfect. Remember that DB Time does not include Oracle background process CPU consumption, yet it is obviously part of the OS CPU utilization.

Summary


My experiment indicated the light-green "CPU wait time" is primarily Oracle foreground process operating system CPU run queue time. This is DB Time "gap" time.

My experiment also showed the "gap" time is highly correlated with CPU utilization. Which means, as the CPU utilization increases, so does the "gap" time.

If there are Oracle Database instrumentation bugs or a host of other potential problems, that will also affect the "gap" time.

If you want a more complete and detailed DB Time formula is would be this:

DB Time = DB CPU + Non Idle Wait Time + gap time

In my next post, I'll show you how to calculate the gap time based on queuing theory!

Thanks for reading!

Craig.








281 comments:

  1. Hi Craig -

    "CPU Wait time is a lousy name because it causes lots of confusion."

    Don't really agree here about the lousy name or the source of confusion, so I wouldn't regard it so much as fact than opinion. Yes the name was decided by committee and maybe it could be better, but "gap time" would not be any less confusing and doesn't easily translate into a relationship with "run queue time". I think the real confusion is due to the fact that this value is derived from other values as you describe, and not measured directly like all the other values on the EM Perf Page. The page should describe the calculation somewhere to eliminate the confusion, not look for a better name.

    One reason this was not called "run queue time" is that it is only a lower-bound estimate of run queue time so we did not want to imply that it was how much run queue time was accumulating altogether. It's important to remember (and you don't mention in your post) that when system is CPU-constrained run queue time will distort the wait event timings, perhaps significantly. That is, some (and maybe lots) of run queue time will be "captured" and accounted as wait time. See my slideshare presentation on ASH for details. This distortion can be really severe in the case of frequent low-latency events like latch gets. If you repeat your experiments with a latch-get-and-CPU-heavy workload and measure the run queue time externally, you should see that the "gap time" only accounts for part of the run queue time, the part when session activity was: CPU-run queue-CPU. When session goes CPU-wait-CPU then the run queue time associated with accessing the CPU after the wait is accounted as part of the wait time (and therefore INCLUDED in DB Time)

    JB

    ReplyDelete
  2. It is really very excellent,i find all articles was amazing.awesome way to get exert tips from everyone,
    not only i like that post all peoples like that post,because of all given information was wonderful and it's very helpful for me.
    dotnet training in Chennai

    ReplyDelete
  3. I do agree with all the ideas you have presented in your post. They’re really convincing and will certainly work. Still, the posts are very short for newbies. Could you please extend them a little from next time? Thanks for the post..

    Web development company in Chennai

    ReplyDelete
  4. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.
    Software Testing Training

    ReplyDelete
    Replies
    1. Really i enjoyed very much. And this may helpful for lot of peoples. So you are provided such a nice and great article within this.

      Hr Consultancy in Chennai

      Delete
  5. Thanks for sharing this information and keep updating us. This is more informatics and it really helped me to know the software Testing.Check the following links about SoftWare Testing .its really helped.


    software Testing Training in Chennai
    testing training in chennai
    Software testing institutes in Chennai

    ReplyDelete
  6. Thanks for sharing the useful information about the topic and good points were stated in the blog for the further information visit the site
    Oracle Fusion Financials Training

    ReplyDelete
  7. Superb. I really enjoyed very much with this article here.
    Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your.

    Oracle Fusion Procurement Online Training

    ReplyDelete
  8. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us.
    Dot Net Training Institute in Chennai

    ReplyDelete
  9. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us.

    Regards:
    Nithya,

    Android Training Institute in Chennai

    ReplyDelete
  10. I would like to say that this blog really convinced me, you give me best information! Thanks, very good post.

    graphics design training in indore
    Keep Posting:)

    ReplyDelete
  11. This comment has been removed by the author.

    ReplyDelete
  12. Whatever We Are Gathering the information from the blogs,we have to keep on updating it,Urgent Care in Chicago.By getting them into one place.Really thanks for posting.Very Thankful for the Informative Post.Really Thanks For Posting.

    ReplyDelete
  13. Great content! Awesome information,Thank you.

    SEO Training in Chennai

    ReplyDelete
  14. Really Thanks For Posting Such an Informative Article....

    Vizag Real Estate

    ReplyDelete
  15. Our digital marketing course in Chennai is targeted at those who are desirous of taking advantage of career opportunities in digital marketing. Marketing professionals who are presently using traditional marketing to meet their business objectives. Start-ups, Entrepreneurs, Business Owners who desire to make use of online media to improve their business goals and objectives.

    Digital Marketing Course in Chennai
    Digital Marketing Training in Chennai
    Online Digital Marketing Training
    SEO Training in Chennai
    Digital Marketing Course
    Digital Marketing Training
    Digital Marketing Courses

    ReplyDelete
  16. This comment has been removed by the author.

    ReplyDelete
  17. This comment has been removed by the author.

    ReplyDelete
  18. Thanks a lot very much for the high quality and results-oriented help. I won’t think twice to endorse your blog post to anybody who wants and needs support about this area
    nebosh course in chennai

    ReplyDelete
  19. Positive site..Really nice to have read..Thanks for sharing..Keep update php training in chennai
    php training in velacher chennai

    ReplyDelete
  20. Thanks for your great and helpful presentation I like your good service.I always appreciate your post.That is very interesting I love reading and I am always searching for informative information like this.Well written article Thank You for Sharing with Us pmp training fee | pmp certification course in chennai | best pmp training institute in chennai| | pmp training class in chennai \ pmp training fee

    ReplyDelete
  21. Thanks for such a great article here. I was searching for something like this for quite a long time and at last, I’ve found it on your blog. It was definitely interesting for me to read about their market situation nowadays pmp training centers in chennai| pmp training in velachery | pmp training near me | project management training in chennai | project management certification online

    ReplyDelete
  22. Thank you for sharing your article. Great efforts put it to find the list of articles which is very useful to know, Definitely will share the same to other forums.

    best openstack training in chennai | openstack course fees in chennai | openstack certification in chennai | redhat openstack training in chennai

    ReplyDelete
  23. Ah,so beautiful and wonderful post!An opportunity to read a fantastic and imaginary blogs.It gives me lots of pleasure and interest.Thanks for sharing.
    Find the Interior Designers in Vijayawada

    ReplyDelete
  24. I really like your wonderful post. I loved to read such kind of article. and the first time I visit our website. and I happy to be here, thanks for sharing this amazing post
    Morgan Digital Marketing

    ReplyDelete
  25. Hello hope you doing well I want to say that your blog is just amazing. Excellent list and genuine information and it works for me. Thank you for sharing I am looking forward to the next blog.
    Digital Marketing Institute in Delhi
    QuickBooks Error 3371

    ReplyDelete
  26. Watch Here Online Pinoy Reply Shows Hd watch latest Pinoy Shows as well as GMA Network Tv episode of Pinoy lambingan
    Ika-5 Utos Pinoy

    ReplyDelete
  27. Thanks for sharing this blog, this blog is very helpful information for every one. You can Watch here Colors Tv Star plus and Zee Tv Episodes
    Watch Internet Wala Love Online

    ReplyDelete
  28. This blog was really amazing, thank you so much for sharing this blog.
    Lifestyle Magazine India

    ReplyDelete
  29. Excellent blog I visit this blog it's really awesome. The important thing is that in this blog content written clearly and understandable. The content of information is very informative. Thanks for the excellent and great idea. keep blogging
    Oracle Fusion Financials Online Training

    ReplyDelete
  30. Thanks for sharing a useful information.. we have learnt so much information from your blog..... keep sharing
    Oracle Fusion HCM Online Training

    ReplyDelete
  31. Excellent blog I visit this blog it's really awesome. The important thing is that in this blog content written clearly and understandable. The content of information is very informative. Thanks for the excellent and great idea. keep blogging
    Oracle Fusion Technical Online Training

    ReplyDelete

  32. Welcome to AZLyrics! It's a place where all searches end! We have a large, legal, every day growing universe of lyrics where stars of all genres and ages shine.

    ReplyDelete
  33. Data Science Certification Courses in Hyderabad. ExcelR is the Best Data Science Training Institute in Hyderabad with Placement assistance and offers a blended.
    https://www.excelr.com/data-science-course-training-in-hyderabad/

    ReplyDelete
  34. Your new valuable key points imply much a person like me and extremely more to my office workers. With thanks; from every one of us.
    apple service center chennai | iphone repair chennai | apple iphone service center in chennai | iphone service chennai

    ReplyDelete
  35. Valuable information provided, thanks for sharing.
    Have got to know about the Stock Market Course in hyderabad from a friend. Interested may checkout the below link- http://avighnastockmarkettraining.com/

    ReplyDelete
  36. Hey Palak! yes, absolutely i have suggested Stock Market Course to other as i see growing career in near future

    ReplyDelete
  37. Thanks for sharing the valuable information with us. will surely recommend this blog to others digital marketing institute

    ReplyDelete
  38. Thanks for sharing this wonderful content with us hope to see more values from you. Also watch this interesting blog Top 10 places to visit in Shimla

    ReplyDelete
  39. Visit for Website Designing & Development Company at Ogen Infosystem.
    Website Designing Company in India

    ReplyDelete
  40. بسم الله الرحمن الرحيم عميلنا العزيز نحن نقدم افضل خدمات تنظيف المجالس والشقق بافضل انواع التنظيفات المشهود لها عالميا
    شركة تنظيف مجالس بالطائف
    شركة تنظيف مجالس بجازان
    شركة تنظيف مجالس بحائل
    ونحن فى خماتكم 24 ساعة على مدار الشهر

    ReplyDelete
  41. Great post. I was once checking constantly this weblog and I'm impressed! Extremely useful information specially the closing part. I maintain such information much. I was once seeking this specific information for a very long time. Many thanks and best of luck.
    apple mobile service centre in chennai
    apple service center near me

    ReplyDelete
  42. Nice blog..! I really loved reading through this article. Thanks for sharing such a amazing post with us and keep blogging... best angularjs training institute in chennai | angularjs training in omr |

    ReplyDelete
  43. Thanks for sharing such a great blog... I am impressed with you taking time to post a nice info.
    App Development Company

    ReplyDelete
  44. I’m going to read this. I’ll be sure to come back. thanks for sharing. and also This article gives the light in which we can observe the reality. this is very nice one and gives indepth information.
    R Training Institute in Chennai | R Programming Training in Chennai

    ReplyDelete

  45. It’s amazing in support of me to truly have a web site that is valuable meant for my knowledge.

    Best Ice Fishing Gloves Best Ice Fishing Gloves Best Ice Fishing Gloves

    ReplyDelete
  46. Truly a very good article on how to handle the future technology. After reading your post,thanks for taking the time to discuss this, I feel happy about and I love learning more about this topic.
    - Mumbai Web, Crocus Web

    ReplyDelete
  47. Great stuff you are sharing!
    love your writing skills.
    keep posting things like this.

    Hindi Panda

    ReplyDelete
  48. Hello,
    Nice article… very useful
    thanks for sharing the information.
    servicenow orchestration training

    ReplyDelete
  49. Well article, interesting to read…
    Thanks for sharing the useful information
    apache training

    ReplyDelete
  50. Thank you so much for given such an informative blog. Get the best Website Designing and Development Services by Ogeninfo.
    Website Designing Company in Delhi

    ReplyDelete
  51. Get the best performing Mutual Fund by Mutual Fund Wala and know the best investment schemes.
    Mutual Fund Agent

    ReplyDelete

  52. نجار بجدة
    تعرف مهنة النجارة بأنها واحدة من أهم المهن الذي يترتب عليها أشياء كثيرة، فالنجار هو من يقوم بصناعة الأثاث بمختلف أنواعه وأشكاله، وهو من يقوم بصناعة الأبواب والنوافذ الخشبية، أيضاً نجار بجدة هو الذي يعمل على فك وتركيب غرف نوم بالمدينة المنورة ويقوم بتركيب دواليب، ومن ضمن المهام الذي يشترك بها نجار بالمدينة المنورة هي عملية نقل الأثاث من مكان إلى آخر، فهو من يعتمد عليه في أعمال فك وتركيب الأثاث وأثاث ايكيا بالمدينة المنورة لأنه يمتلك قدرات وإمكانيات كبيرة تساعده للقيام بذلك وأكثر، يحرص معلم نجار بالمدينة المنورة على استخدام الأدوات اليدوية والأدوات التي تستخدم بواسطة الكهرباء وجميعها مهمة بالنسبة له وتساعد في تنفيذ كافة أعماله.

    ReplyDelete
  53. Nice blog, Visit Kalakutir Pvt Ltd for Godown Line Marking Painting, Base Company Logo Painting, and School Bus Painting.
    Base Company Logo Painting

    ReplyDelete

  54. Thanks for sharing nice information with us. I like your post and all you share with us is up to date and quite informative,

    Thanks
    Cpa offers


    ReplyDelete
  55. I am impressed by the information that you have on this blog amazon web services training . It shows how well you understand this subject.

    ReplyDelete
  56. Thanks for this amazing post its help me a lot to solve my issues click here to download Kinemaster Mod APK

    ReplyDelete
  57. I see some amazingly important and kept up to length of your strength searching for in your on the siteAmazon web services bangalore

    ReplyDelete
  58. Usually I never comment on blogs but your article is so convincing that I never stop myself to say something about it. You’re doing a great job Man,Keep it up.
    study in uk from india

    ReplyDelete
  59. Buys online can be made utilizing Visa or MasterCard. Customers can return thing for trade or discount if not happy with quality. For more data or questions on installment, discount, crossing out, or others, achieve the Cleveland Browns bolster. visit here
    https://www.vodafone-customer-care-number.in/up-east/

    ReplyDelete
  60. Thanks For your post.

    clipping path

    For the globalization of internet & web based foundation it has become so near to reach the Door to Door to provide all kinds of support & service. Clipping Path EU is a company of web & internet base image editing service Provider Company which provides all kinds of image treatment. If you are familiar with this you will know the services like Photoshop Clipping Path, Photo Retouching, Image Masking, Color Correction, Photo Restoration, and Logo Design, Raster 2 Vector works, image editing and also Photoshop editing service.

    ReplyDelete
  61. amazing post and written in very simple and impressive language. Thanks for sharing
    harry potter wifi names

    ReplyDelete
  62. Nice Post. You have done a great job. Please Keep Posting and Keep Sharing. Emotional Quotes

    ReplyDelete
  63. Technical Bagle a best resource to get all kinds of Knowledge about Wordpress, Blogging, SEO, Social Media, YouTube and Make Money Online.

    how to recover deleted whatsapp chat

    ReplyDelete
  64. Thanks For sharing a nice post about AWS Training Course.It is very helpful and AWS useful for us.aws training in bangalore

    ReplyDelete
  65. It is very good and useful for students and developer .Learned a lot of new things from your post!Good creation ,thanks for give a good information at AWS.aws training in bangalore

    ReplyDelete
  66. thank you very much for share this wonderful article 토토사이트

    ReplyDelete
  67. thank you very much for share this wonderful article 토토사이트

    ReplyDelete
  68. Thanks for sharing, nice post! Post really provice useful information!

    Hương Lâm chuyên cung cấp bán máy photocopy và dịch vụ cho thuê máy photocopy giá rẻ, uy tín TP.HCM với dòng máy photocopy toshiba và dòng máy photocopy ricoh uy tín, giá rẻ.

    ReplyDelete
  69. Thanks for sharing, nice post! Post really provice useful information!

    Hương Lâm chuyên cung cấp bán máy photocopy và dịch vụ cho thuê máy photocopy giá rẻ, uy tín TP.HCM

    ReplyDelete
  70. Thanks To sharing blog Excellent information with us...

    sparkinfosys.com

    Sparkinfosys is one of the best company in hyderabad ,We provide products are
    E-commerce B2c ,E-commerce B2cB, Online Exam,Matrimonial,Classified,we designing company Logo Designing,Domain Registration etc..

    ReplyDelete
  71. More impresiive Blog!!! Its more useful for us...Thanks for sharing with us...
    ccc online paper

    ReplyDelete
  72. Thanks for the information...
    AWS Training in Bangalore | AWS Cours | AWS Training Institutes - RIA Institute of Technology
    - Best AWS Training in Bangalore, Learn from best AWS Training Institutes in Bangalore with certified experts & get 100% assistance.

    ReplyDelete
  73. I seriously love your site.. Excellent colors & theme. Did you create this website yourself? Please reply back as I’m trying to create my own personal site and would love to know where you got this from or exactly what the theme is called. Cheers!
    Techno

    ReplyDelete
  74. Hey, Wow all the posts are very informative for the people who visit this site. Good work! We also have a Website. Please feel free to visit our site. Thank you for sharing. Well written article Thank You for Sharing with Us

    ReplyDelete
  75. Awesome article, it was exceptionally helpful! I simply began in this and I'm becoming more acquainted with it better! Cheers, keep doing awesome
    Matsya University BCOM TimeTable 2020

    ReplyDelete
  76. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. Keep up the good work!


    data analytics course

    data science course

    business analytics course

    ReplyDelete
  77. Thanks To sharing blog Excellent information with us...
    sparkinfosys
    Sparkinfosys is one of the best company in hyderabad ,We provide Features of online Examination website designed and developed by Spark Infosys Hyderabad. Spark Infosys provides online exam portal development etc..

    ReplyDelete
  78. Thanks for sharing such a great information..Its really nice and informative..

    best aws training in bangalore
    aws online training

    ReplyDelete
  79. Thanks for sharing valuable information.It will help everyone.keep Post.

    Web designing trends in 2020

    When we look into the trends, everything which is ruling today’s world was once a start up and slowly begun getting into. But Now they have literally transformed our lives on a tremendous note. To name a few, Facebook, Whats App, Twitter can be a promising proof for such a transformation and have a true impact on the digital world.

    we have offered top the advacned syllabus course web design and development for available join now.

    more details click the link now.

    https://www.webdschool.com/web-development-course-in-chennai.html

    ReplyDelete
  80. Thanks for the interesting blog that you have implemented here. Very helpful and innovative. Waiting for your next upcoming article.
    Java training in chennai
    Java training institute in chennai
    Java course in chennai
    Java training classes
    Java training
    Java programming classes
    core java coure
    <a

    ReplyDelete
  81. Thanks for the interesting blog that you have implemented here. Very helpful and innovative. Waiting for your next upcoming article.
    Java training in chennai
    Java training institute in chennai
    Java course in chennai
    Java training classes
    Java training
    Java programming classes
    core java coure
    <a

    ReplyDelete
  82. Very Nice and great This was an excellent post and very good information provided,Thanks for sharing 더킹카지노.

    ReplyDelete
  83. Thanks For Sharing Really Helpfull Post Diabetes Ke Kaarn Thanks For Sharing Really Helpfull Post <a

    ReplyDelete

  84. That is nice article from you , this is informative stuff . Hope more articles from you . I also want to share some information about redhat openstack training and mainframe tutorial videos

    ReplyDelete
  85. Thanks for provide great informatic and looking beautiful blog, really nice required information & the things i never imagined and i would request 바카라사이트

    ReplyDelete
  86. I would really like to read some personal experiences like the way, you've explained through the above article. I'm glad for your achievements and would probably like to see much more in the near future. Thanks for share.

    ReplyDelete
  87. I have to agree with the valid points you make in your article because I see things like you. Additionally, your content is interesting and really good reading material. Thank you for sharing your talent.
    SAP training in Kolkata
    SAP training Kolkata
    Best SAP training in Kolkata
    SAP course in Kolkata
    SAP training institute Kolkata

    ReplyDelete
  88. Your Writer in write Very Nice Article this topic Thanks A lot shareing This Important Information
    Pradhan mantri awas yojana gramin

    ReplyDelete
  89. https://thesecondgradesuperkids.blogspot.com/2015/10/designating-space-for-reading-with.html?showComment=1593686069690#c3527332811773302338

    ReplyDelete
  90. https://thesecondgradesuperkids.blogspot.com/2015/10/designating-space-for-reading-with.html?showComment=1593686069690#c3527332811773302338

    ReplyDelete
  91. https://thesecondgradesuperkids.blogspot.com/2015/10/designating-space-for-reading-with.html?showComment=1593686069690#c3527332811773302338

    ReplyDelete
  92. https://www.adaptflex.com/

    Your article is very informative. Thanks for sharing it!

    ReplyDelete
  93. I have recently visited your blog profile. I am totally impressed by your blogging skills and knowledge.

    SAP Online Training

    SAP Classes Online

    SAP Training Online

    Online SAP Course

    SAP Course Online

    ReplyDelete
  94. Hi I am Siya, I am providing the Home tuition in Sahibabad. Hometutorials.net also provide the Home tution in Sahibabad. Many parents find that their children are more receptive in working through their studies with a teacher at home.

    ReplyDelete
  95. Hi, I am Stella brown I am reading your blog, It is an amazing blog thanks for the sharing the blog. Allassignmentservices.com provide the marketing management assignment such as an information about education for students queries. Allassignmentservices also provide the Artificial Intelligence Assignment Help. Students from different walks of life can afford our assignment help services as we are affordable. Moreover, our team of academic writers is highly efficient at their work.

    ReplyDelete
  96. This article is very helpful for me. Visit This Website formore solutions
    JASHABHSOFT

    ReplyDelete
  97. If you are passing through this technical issue, our online Garmin professionals are very proficient for clearing your doubts instantly. We are available 24/7. Thank you so much for sharing this great blog.Very inspiring and helpful too.Hope you continue to share more of your ideas. I will definitely love to read. Know More About Garmin.com/Express.

    ReplyDelete
  98. Its a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much
    DevOps Training in Chennai

    DevOps Course in Chennai

    ReplyDelete
  99. Garmin.com/Express is the final step of getting your Garmin application in your device. However, there are quite a few other steps that you will need to perform before installing Garmin. We are available 24/7!

    ReplyDelete
  100. blue prism training in Chennai - If you choose to learn the blue prism or automation tool you are supposed to have the programming language. start to learn the blue prism training from the Best Blue prism Training Institute in Chennai.

    uipath training in Chennai - UI path technology is one of the fastest developing fields which has a lot of job opportunities such as software developer, Programmer and lot more. Join the Best Uipath Training Institute in Chennai.

    microsoft azure training in chennai -Microsoft azure technology is growing and soon it will be competitive aws. So students who start to learn Microsoft azure now will be well - paid in the future. Start to learn Microsoft azure training in Chennai.

    Chennai IT Training Center

    ReplyDelete
  101. I think this is among the most vital information for me. And i am glad reading your article.
    Thanks!
    visit my sites Please.

    1) http://www.coffeeincoffee.com/bbs/board.php?bo_table=m42&wr_id=817&page=0&sca=&sfl=&stx=&sst=&sod=&spt=0&page=0
    2) http://gc.whitskin.com/gnuboard4/bbs/board.php?bo_table=lnc_review&wr_id=243699&page=0&sca=&sfl=&stx=&sst=&sod=&spt=0&page=0
    3) http://www.xn--hz2bi2lk7d36am1as6bdy4bkgd.kr/bbs/board.php?bo_table=yp_board02&wr_id=78
    4) http://www.kk3921.kr/main/bbs/board.php?bo_table=s308_2&wr_id=611980
    5) http://www.ojmflower.co.kr/shop_goods/review_read.htm?review_idx=1043

    ReplyDelete
  102. valuable blog,Informative content...thanks for sharing, Beard Transplant

    ReplyDelete
  103. Looking for best English to Tamil Typing online, make use of our site to enjoy Tamil typing and directly share on your social media handle. Tamil typing software free download

    ReplyDelete
  104. Great Article, Thanks for the nice & valuable information. Here I have a suggestion that if your looking for the Best Digital Marketing Institute in Rohini Then Join the 99 Digital Academy. 99 Digital Academy offers an affordable Digital Marketing Course in Rohini. Enroll Today.

    ReplyDelete
  105. Thanks for sharing.
    I found a lot of interesting information here.
    A really good post, very thankful
    and hopeful that you will write many more posts like this one.

    नरेगा जॉब कार्ड

    Shala Darpan

    Govt.Schemes

    Fantastic Game

    ReplyDelete
  106. Great post. Sailpoint training in Canada is a best institute for training.

    ReplyDelete