Friday 26 April 2013

When you make something, cleaning it out of structural debris is one of the most vital things you do.

Collabora recently had a problem with a project's ARM build farm. In a nice change of pace it was not that the kernel was crashing, nor indeed any of the software or hardware.

The Problem

Instead our problem was our build farm could best be described as "a pile of stuff" and we wanted to add more systems to it and have switched power control for automated testing.

Which is kinda where the Christopher Alexander quote comes into this. I suggested that I might be able to come up with a better, or at least cleaner, solution.

The Idea

A subrack with sub modulesPrevious experience had exposed me to the idea of using 19 inch subracks for mounting circuits inside submodules.

I originally envisaged the dev boards individually mounted inside these boxes. However preliminary investigation revealed that the enclosures were both expensive and used a lot of space which would greatly increase the rack space required to house these systems.

imx53 QSB eurocard carrier
I decided to instead look at eurocard type subracks with carriers for the systems. Using my 3D printer I came up with a carrier design for the imx53 QSB and printed it. I used the basic eurocard size of 100mm x 160mm which would allow the cards to be used within a 3U subrack.

Once assembled it became apparent that each carrier would be able to share resources like power supply, ethernet port and serial console via USB just as the existing setup did and that these would need to be housed within the subrack.

The Prototype

The carrier prototype was enough to get enough interest to allow me to move on to the next phase of the project. I purchased a Schroff 24563-194 subrack kit and three packs of guide rails from Farnell and assembled it.

Initially I had envisaged acquiring additional horizontal rails from Schroff which would enable constructing an area suitable for mounting the shared components behind the card area.

Rear profile for Schroff subrackUnfortunately Schroff have no suitable horizontal profiles in their catalog and are another of those companies who seem to not want to actually sell products to end users but rather deal with wholesalers who do not have their entire product range!

Printed rear profile for Schroff subrack
Undaunted by this I created my own horizontal rail profile and 3D printed some lengths. The profile is designed to allow a 3mm thick rear cover sheet attached with M2.5 mounting bolts and fit rack sides in the same way the other profiles do.

At this point I should introduce some information on how these subracks are dimensioned. A standard 19 inch rack (as defined in IEC 60297) has a width of 17.75 inches(450.85mm) between the vertical posts. The height is measured in U (1.75 inches)

A subrack must obviously fit in the horizontal gap while providing as much internal space as possible. A subrack is generally either 3 or 6 U high. The width within a subrack is defined in units called HP (Horizontal Pitch) which are 0.2 inches(5.08 mm) and subracks like the Schroff generally list 84 usable HP.

However we must be careful (or actually just learn from me stuffing this up ;-) as the usable HP is not the same thing as the actual length of the horizontal rails! The enclosures actually leave and additional 0.1 inch at either end giving a total internal width of 85HP (17 inches, 431.8 mm) which leaves 0.75 inches for the subrack sides and some clearance.

The Schroff subrack allows eurocards to be slotted into rails where the card centre line is on HP boundaries, hence we describe the width of a card in the slot in terms of HP

I cannot manufacture aluminium extrusions (I know it is a personal failing) nor produce more than 100 mm long length of the plastic profile on my printer.

Even if full lengths are purchased from a commercial service (120 euros for a pair including tax and shipping) the plastic does not have sufficient mechanical strength.

The solution I came up with was somewhat innovative, as an alternative a M5 bolt into a thread in the aluminium extrusion I used a 444mm long length of 4mm threaded rod with nuts at either end. This arrangement puts the extrusion under compression and gives it a great deal of additional mechanical strength as the steel threaded rod is very strong.

Additionally to avoid having to print enough extrusion for the entire length I used some 6mm aluminium tube as a spacer between 6HP(30.48mm) wide sections of the printed extrusion.

It was intended to use a standard modular PC power supply which is 150mm wide which is pretty close to 30HP (6 inches) so it was decided to have a 6HP section of rail at that point to allow a rear mounting plate for the PSU to be attached.

This gives 6HP of profile, 21HP(106.68mm) of tube spacer, 6HP of profile, 46HP(233.68 mm) of tube spacer and a final 6HP profile summing to our total of 85HP. Of course this would be unnecessary if a full continuous 85HP rail had been purchased, but 6 of 6 HP long profile is only 51 euro a saving of 70 euro.

To provide a flat area on which to mount the power switching, Ethernet switch and USB hubs I ordered a 170 x 431 mm sheet of 3mm thick aluminium from inspiredsteel who, while being an ebay company, were fast, cheap and the cutting was accurate.

Do be sure to mention you would prefer it if any error made the sheet smaller rather than larger or it might not fit, for me though they were accurate to the tenth of a mm! If you would prefer the rear section of the rack to be enclosed when you are finished, buy a second sheet for the top. For my prototype I only purchased a 170 x 280mm sheet as I was unsure if I wanted a surface under the PSU (you do, buy the longer sheet)

PC power supply mounted to back plateMounting the PSU was a simple case of constructing a 3 mm thick plate with the correct cutouts and mounting holes for an ATX supply. Although the images show the PSU mounted on the left hand side of the rack this was later reversed to improve cable management.

The subrack needed to provide Ethernet switch ports to all the systems. A TP-Link TL-SF1016DS 16-Port 10/100Mbps Switch  was acquired and the switch board removed from its enclosure. The switch selected has an easily removed board and is powered by a single 3.3V input which is readily available from the ATX PSU.

Attention now returned to the eurocard carriers for the systems, the boards to be housed were iMX53 QSB and iMX6 SABRE Lite and a Raspberry Pi control system to act as USB serial console etc.

The carriers for both main boards needed to be 8HP wide, comprised of:
  • Combined USB and Ethernet Jack on both boards was 30 mm tall 
  • PCB width of 2mm
  • underside components of 4mm
  • clearance between boards of 2mm
Although only 38 mm this is 7.5HP and fractions of an HP are not possible with the selected subrack.

With 8HP wide modules this would allow for ten slots, within the 84 usable HP, and an eleventh 4HP wide in which the Raspberry Pi system fits.

iMX6 SABRE Lite eurocard carrierCarrier designs for both the i.MX53 QSB and the i.MX6 SABRE Lite boards were created and fabricated at a professional 3D print shop which gave a high quality finish product and removed the perceived risk of relying on a personal 3D printer for a quantity of parts.

This resulted in changes in the design to remove as much material as possible as commercial 3D services charge by the cubic cm. This Design For Manufacture (DFM) step removed almost 50% from the price of the initial design. 

i.MX53 QSB carriers with wiring loom
The i.MX6 design underwent a second iteration to allow for the heatsink to be mounted and not mechanically interfere with the hard drive (although the prototype carrier has been used successfully for a system that does not require a hard drive). The lesson learned here is to be aware that an design iteration or two is likely and that it is not without cost.

The initial installation was to have six i.MX53 and two i.MX6 this later changed to a five/four split, however the carrier solution allows for almost any combination, the only caveat (discovered later) is the imx53 carriers should be to the right hand side with the small 4HP gap at that end as they have a JTAG connector underneath the board which otherwise foul the hard drive of the next carrier.

Racked cards showing unwanted cable tails
A wiring loom was constructed for each board giving them a connector tail long enough to allow them to be removed. This was the wrong approach! if you implement this design (or when I do it again) the connector tails on the wiring loom should present all the connections to the rear at the same depth as the Ethernet connection.

The rack cables themselves should be long enough to allow the slides to be removed but importantly it is not desirable to have the trailing cable on the cards. I guess the original eurocard designers figured this out as they designed the cards around the standard fixed DIN connectors at the back of the card slots.

USB relay board with wiring loom attached
We will now briefly examine a misjudgement that caused the initially deployed solution to be reimplemented. As the design was going to use USB serial converters to access the serial console a USB connected relay board was selected to switch the power to each slot. I had previously used serial controlled relay boards with a USB serial convertor however these were no longer available.

Initial deployment with USB controlled relay board
All the available USB relay boards were HID controlled, this did not initially seem to be an issue and Linux software was written to provide a reasonable interface. However it soon became apparent that the firmware on the purchased board was very buggy and crashed the host computer's USB stack multiple times.

Deployed solution

Once it became apparent that the USB controlled power board was not viable a new design was conceived. As the Ethernet switch had ports available Ethernet controlled relay boards were acquired.

Evolution of 3mm PCB pillars
It did prove necessary to design and print some PCB support posts with M3 nut traps to allow the relay boards to be easily mounted using double sided adhesive pads.

By stacking the relay boards face to face and the Ethernet switch on top separated using nylon spacers it was possible to reduce the cable clutter and provide adequate cable routing space.

A busbar for Ground (black) and unswitched 12V (yellow) was constructed from two lengths of 5A chock block.

An issue with power supply stability was noted so a load resistor was added to the 12V supply and an adhesive thermal pad used to attach it to the aluminium base plate.

Completed redesign
It was most fortunate that the ethernet switch mounting holes lined up very well with the relay board mounting holes allowing for a neat stack.

This second edition is the one currently in use, it has proved reliable in operation and has been successfully updated with additional carriers.

The outstanding issues are mainly centered around the Raspberry Pi control board:
  • Needs its carrier fitting. It is currently just stuck to the subrack end plate.
  • Needs its Ethernet cable replacing. The existing one has developed a fault post installation.
  • Needs the USB hub supply separating from the device cable. The current arrangement lets the hub power the Pi which means you cannot power cycle it.
  • Connect its switched supply separately to the USB hub/devices.

Shopping list

The final bill of materials (excluding labour and workshop costs) which might be useful to anyone hoping to build their own version.

Prices are in GBP currency converted where appropriate and include tax at 20% and delivery to Cambridge UK and were correct as of April 2013.

The purchasing was not optimised and for example around 20GBP could be saved just by ordering all the shapeways parts in one order.
Base subrack
ItemSupplierQuantityLine Price
Schroff 24563-194 subrack kitFarnell141.28
Schroff 24560-351 guide railsFarnell313.65
Schroff rack rear horizontal railShapeways2100.00
1000mm length of 4mm threaded rodB and Q11.48
170mm x 431mm x 3mm Aluminium sheetinspired steel240.00
PSU mounting plateShapeways135.42
PCB standoffShapeways422.30
160mm Deep Modular PC supplyCCL155.76
TP-Link TL-SF1016DS 16-Port 10/100Mbps-SwitchCCL123.77
8 Channel 16A Relay Board Controlled Via EthernetRapid2126.00
Raspberry PiFarnell126.48
USB Serial convertersCCL1037.40
10 port strip style USB HUBEbay17.00
Parts for custom Ethernet cablesRS1326.00
Parts for custom molex power cables (salvaged from scrap ATX PSU)Workshop1111.00
33R 10W wirewound resistor for dummy loadRS11.26
24pin ATX female connector pre-wiredMaplin12.99
Akasa double sided thermal padMaplin15.00
Small cable tie basesMaplin16.49
Miscellaneous cable, connectors, nylon standoffs, solder, heatshrink, zip ties, nuts, washers etc. Workshop120.00
Total for subrack603.28

The carriers are similarly not optimally priced as over five GBP each can be saved by combining shipping on orders alone. Also the SSD drive selection was made some time ago and a newer model may be more suitable.
i.MX53 QSB carrier
ItemSupplierQuantityLine Price
i.MX53 QSBFarnell1105.52
Intel 320 SSD 80GCCL1111.83
Carrier boardShapeways130.00
combined sata data and power (15 to 20cm version)EBay15.00
Low profile right angle 5.5mm x 2.1mm barrel jackEBay10.25
Parts for 9pin serial cable extensionRS15.00
Miscellaneous solder, heatshrink, nylon nuts, bolts and washersWorkshop15.00
Total for carrier262.60

i.MX6 SABRE Lite carrier
ItemSupplierQuantityLine Price
i.MX6 SABRE LiteFarnell1128.06
Intel 320 SSD 80GCCL1111.83
Carrier boardShapeways135.00
combined sata data and power (15 to 20cm version)EBay15.00
Low profile right angle 5.5mm x 2.1mm barrel jackEBay10.25
Parts for 9pin serial cable modificationRS12.00
Miscellaneous solder, heatshrink, nylon nuts, bolts and washersWorkshop15.00
Total for carrier287.14

Conclusion

The solution works and in a 3U high 355mm deep subrack ten ARM development boards can be racked complete with local ethernet switching, power control and serial consoles.

Deployed system in situ configured as a build and test farm
The solution is neat and provides flexibility, density and reproducibility the "pile of stuff" solution failed to do.

For current prototype with nine filled slots the total cost was around 3000GBP or around 330GBP per slot which indicates a 100GBP per slot overhead over the "pile of stuff" solution. These figures omit the costs of the engineer and  workshop time, which are estimated at an additional 1500GBP. Therefore a completed rack, fully filled with i.MX6 carriers costs around 5000GBP

Density could be increased if boards with lower height requirements were used however above twelve units there issues with Ethernet switch, power switch and USB port availability become a factor. For Example the 16 port Ethernet switch requires a port for uplink, one for each relay board and one for the console server which leaves only 12 ports for systems.

Addressing the outstanding issues would result in a much more user friendly solution. As the existing unit is in full time use and downtime is not easily scheduled for all ten systems, the issues are not likely to be fixed on the prototype and would have to be solved on a new build.

The solution is probably not suitable for turning into a product but that was not really the original aim. A commercial ARM blade server using this format would almost certainly use standard DIN connectors and a custom PCB design rather than adapting existing boards.

49 comments:

  1. A cheaper option to the ethernet relays is a sainsmart relay board driven directly by the raspberry GPIO pins:

    https://coderwall.com/p/izzsig

    ReplyDelete
    Replies
    1. As I understand it the Pi does not have the 16 GPIO pins that would be necessary to drive the 16 relay ports without an additional I/O expander.

      Given how much trouble we had with the USB relay board the Ethernet relay board are really not the principle expenditure.

      I also chose to document what we have actually working as a soilution rather than speculate of alternative solutions.

      Delete
  2. This comment has been removed by a blog administrator.

    ReplyDelete
  3. Hi !
    Very nice setup !
    What is the power usage at load and at idle ?
    Can you turn on/off any particular board ?

    thanks !

    ReplyDelete
    Replies
    1. The power load is around 70W total at all times, load has little difference.

      The Ethernet switch, two relay boards and the load resistor use 20W of that and the remaining 50W is used by the 10 systems.

      Do not know what the individual breakdown is as some have hard drives and some not.

      And yes the relay boards provide complete Ethernet controlled power sequencing for all 11 slots and the USB hub.

      Delete
  4. This comment has been removed by a blog administrator.

    ReplyDelete
  5. Thanks for referencing our Devontech relay board. Have you thought of leaving a review and maybe getting £25 off your next Rapid order?

    ReplyDelete
  6. Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I'll be subscribing to your feed and I hope you post again soon. Big thanks for the useful info.
    office cleaning richmond

    ReplyDelete
  7. There is some debate about the best blog host to use for your blog. However when it comes to choosing a blog host with a wide variety of options.

    Floor Sealing
    Line Marking

    ReplyDelete
  8. كل هذة الخدمات مقدمة من شركة ركن نجد كذلك خدمات شركة تنظيف بجدة معتمدين علي الكفائة المهنية التي نتشرف بتقديمها لكم عملائنا الكرام وكذلك المعدات والماكينات الحديثة التي تمتلكها الشركة في سعينا الدائم لكي نكون افضل شركة تنظيف بجدة .
    جميع خدمات  شركة ركن نجد شركة تنظيف منازل بجدة مضمونة وتستخدم افضل تقنيات ومعدات التنظيف وكذلك مواد التنظيف المستخدمة لا تتسبب ابدا في اية اضرار للرخام الاسباني او احواض المياه اليطالي والتي اذا تم استخدام الاسيد عليها يتم افسادها وافساد الشكل العام لها وكذلك خدمات التنظيف المميزة تشمل :

    ReplyDelete

  9. شركة فارس الخليج
    افضل شركة خدمات منزلية بالمملكة العربية السعودية
    شركة فارس الخليج شركة رسمة ذات ثقة
    تتميز شركتنا بامتلاك افضل طاقم عمل لاداء الخدمات المنزلية
    نستخدم افضل الاجهزة الموجودة.
    شركه تنظيف بعنيزة
    شركه تنظيف ببريدة
    شركه رش مبيدات ببريدة
    شركه تنظيف خزانات ببريدة
    افضل شركه رش مبيدات ببريدة

    ReplyDelete
  10. This comment has been removed by the author.

    ReplyDelete
  11. This comment has been removed by the author.

    ReplyDelete
  12. Thanks for sharing informative post. Looking for best window cleaning Sunshine coast, we are the professional cleaning company offering window cleaning, bond cleaning & other cleaning services.

    ReplyDelete
  13. Thank you for this post.This is very interesting information for me. argumentative essay thesis.

    ReplyDelete
  14. Thanks for this information, I think it will come in handy in the future. I know how to focus on homework - https://homeworkfor.me/blog/compare-and-contrast-essay-tips-for-students .

    ReplyDelete
  15. Thanks for sharing informative post. Looking for best commercial cleaners Sunshine coast, we are the professional cleaning company offering professional cleaning services.

    ReplyDelete
  16. If you can relate to any of these problems, write do my homework on our live chat portal. Let our brilliant team of writers handle the pressure for you.

    ReplyDelete

  17. Class College Education training Beauty teaching university academy lesson  teacher master student  spa manager  skin care learn eyelash extensions tattoo spray

    ReplyDelete
  18. Test https://infogram.com/dang-ky-kubet-cach-tao-lap-tai-khoan-ku-bet-don-gian-1h8j4xglr0od6mv

    ReplyDelete
  19. Taxitaithanhhung.vn - công ty cung cấp giải pháp chuyển nhà trọn gói tại Hà Nội. Với mong muốn mang đến cho quý khách hàng dịch vụ vận chuyển nhà tại Hà Nội tốt nhất. Đội ngũ nhân viên chuyển nhà trọn gói Thành Hưng chuyên nghiệp, hệ thống xe tải với số lượng lớn, chúng tôi tự tin cam kết mang đến cho quý khách hàng 1 dịch vụ tốt nhất!

    Từ khóa chuyển nhà Hà Nội - công ty cung cấp dịch vụ chuyển nhà Thành Hưng:

    #chuyennhahanoi #chuyennhataihanoi #dichvuchuyennhahanoi #dichvuchuyennhataihanoi #dichvuchuyennhathanhhung #chuyennhatrongoithanhhung #chuyennhatrongoihanoi #chuyennhatrongoitaihanoi #donnhathanhhung #donnhatrongoihanoi #chuyennhagiarehanoi #dichvuchuyennhagiarehanoi

    https://tuoitre.vn/chuyen-nha-thanh-hung-20-nam-dong-hanh-va-phat-trien-thuong-hieu-20200422112222106.htm

    https://vnexpress.net/20-nam-taxi-tai-thanh-hung-phuc-vu-khach-hang-4075831.html

    https://dantri.com.vn/kinh-doanh/dich-vu-chuyen-nha-tron-goi-thanh-hung-20200319160451993.htm

    https://filmfreeway.com/chuyennhatrongoithanhhung

    https://medium.com/@vantaitthanhhung/dich-vu-chuyen-nha-thanh-hung-tai-ha-noi-9ddbff5cda4

    ReplyDelete
  20. While providing dissertation writing services to students, now all I need is a harvesting tool that cut the vegetables itself and need a robot that cook these vegetables.

    ReplyDelete
  21. Taxitaithanhhung.vn- công ty cung cấp dịch vụ chuyển nhà trọn gói tại Hải Phòng - chuyển nhà tại Hải Phòng - chuyển nhà Hải Phòng giá rẻ.

    https://medium.com/@vantaitthanhhung/chuyen-nha-tron-goi-hai-phong-taxitaithanhhung-vn-ed41a7a9c6f0

    https://sketchfab.com/chuyennhahaiphong

    https://infogram.com/dich-vu-chuyen-nha-tron-goi-hai-phong-1h7j4d83vp0x6nr?live

    https://stocktwits.com/chuyennhahaiphong

    https://www.domestika.org/en/chuyennhahaiphong

    Website chuyển nhà Thành Hưng: Taxitaithanhhung.vn

    Từ khóa chuyển nhà:
    #taxitaithanhhung
    #chuyennhahaiphong
    #chuyennhatrongoihaiphong
    #chuyennhataihaiphong
    #chuyennhahaiphonggiare
    #dichvuchuyennhahaiphong
    #vanchuyennhahaiphong
    #chuyennhahanoiuytin
    #dichvuchuyennhatrongoihaiphong

    ReplyDelete
  22. Struggling with Quickbooks Support Phone Number ? Fix it by dialling us at 1-855-533-6333. The error usually arrives due to improper installation of QuickBooks upgrading.

    ReplyDelete
  23. Nice Blog !
    Looking for comprehensive assistance while working on QuickBooks? No worries!! Just reach out to our executives by dialing our QuickBooks Phone Number 1-888-927-O94O.

    ReplyDelete
  24. Wonderful Post, I highly appreciate those people who share some good information.
    Technical Hub
    Yola Website
    Technical Hub
    Technical Institute

    ReplyDelete
  25. Thanks for sharing excellent informations. Your web-site is very cool. I’m impressed by the details that you have on this site. It reveals how nicely you understand this subject. Bookmarked this web page, will come back for more articles. You, my pal, ROCK! I found simply the information I already searched all over the place and simply couldn’t come across. What a great web-site.

    office 2019 setup

    ReplyDelete
  26. Advertising with us allows you to reach a very large audience in Sindhi, Pushto and other regional languages. We offer daily Kawish Advertisement and also operating as Aaj Peshawar advertising agency.
    office automation services
    hearing aid Pakistan
    hearing aids in lahore
    custom suits
    ad agency
    chemical industries in lahore
    Architecture Designs in lahore
    mbbs in china
    hearing clinic in lahore


    ReplyDelete
  27. Do you need help with issues you are facing in QuickBooks? If so!! Then connect with our experts at Quickbooks Customer Service Phone Number USA | Canada +1-855-929-0120. and eliminate the obstacles to your workflow. They are available 24/7 with value for money services!!

    QuickBooks Support Phone Number +1-855-929-0120
    Quickbooks Customer Service Phone Number | Quickbooks Support +1-855-929-0120
    QuickBooks Support Phone Number California USA +1-855-526-0297
    QuickBooks Technical Support Phone Number Nevada +1-855-526-0297

    ReplyDelete
  28. كثيراً ما يضيّع الإنسان الكثير من وقته في قراءة كتاب غير مفيد، أو قراءة كتاب صعب بينما هناك الأسهل، أو كتاب سطحي بينما هناك الأعمق.
    مدافن للبيع
    مقابر للبيع
    بي فلاي

    ReplyDelete
  29. Also, because of this problem, you will not be able to send text messages or connect to the Internet unless you use Wi-Fihttps://khabarpu.com/rp/1628052521.htm

    ReplyDelete
  30. While searching for the best spot to fix your QB errors, you can approach to manage our QuickBooks Customer Support Phone Number - Colorado USA +1-833-899-5884., and our experts will give you the best service. Our expert group will help you fix and research your issues immediately. You can call us at Quickbooks Enterprise Support Number today to fix your goofs.

    ReplyDelete

  31. It's been nice to see such types of amazing content!!
    I would Love to share
    Home Cleaning in Hyderabad

    ReplyDelete
  32. Our QuickBooks Customer Service pack is reliably ready and prepared to give you the best solution for your errors. Call us at our QuickBooks Customer Support and get the right solution. Presently, you need capable help by dialing our QuickBooks Customer Support Service Phone Number - - Texas +1-888-293-0848 to sort this issue. Our team is accessible 24*7 in your service to help resolve your errors.

    ReplyDelete
  33. Make an effort not to spare a second to speak with our expert QB trained professionals. We can help you and tackle your issues. call us at QuickBooks Customer Support Phone Number - Florida USA +1-888-893-3387 today and get the best solution.

    ReplyDelete
  34. In the event that you are seeking for the expert assistance for your QB software, you can talk with our experts at QuickBooks Customer Support Phone Number - Montana USA +1-406-226-7046 today.

    ReplyDelete
  35. Need second QB support? Lets partner with us at QuickBooks Customer Support Phone Number - Washington DC USA +1-844-297-7671 today. We can help with resolving your issues and give you the best solution.

    ReplyDelete
  36. Communicate with professionals at QuickBooks Enterprise Support. You can connect with our experts through QuickBooks Customer Service +1-888-865-6485 in order to fix many QuickBooks related errors.

    ReplyDelete
  37. What is Kisho Ino digital currency and why should we invest in it? Talking about digital currencies is as exciting as buying one of these currencies in the market. Especially if بازار کیشو اینو در تمام صرافی های جهان we are going to talk about one of the most popular digital currencies in the Iranian market these days, namely Kisho Ino.

    ReplyDelete
  38. This comment has been removed by the author.

    ReplyDelete
  39. Great blog and Very useful information on cleaning services and looking for best cleaning Services in Delhi India.

    ReplyDelete
  40. Thank You for sharing such an Amazing article!! I would love to share in Deep cleaning Services in Hyderabad

    ReplyDelete