Affecting System - Client Billing/Support Portal
Here are the details of the scheduled maintenance:
Date: May 3, 2024 Time: 11:00 PM EST Duration: 60 Minutes
Billing/Support system for Dedispec will be intermittently down during this time. This will also affect any replies sent for existing or new Support tickets. It would be recommended to wait until after this maintenance to send any emails for new or existing tickets.
No client servers will be affected as our Billing/Support system is hosted at a separate datacenter location where no client servers are housed.
Affecting Other - Power
We've experienced a cascading power failure on both our A and B power feeds.
Our crews are working to restore power and we will update you again shortly.
We’ll provide updates here as they are available.
Thank you
Update:
As of approx 6PM EST, power at the Kansas City, MO datacenter location has been restored. If you continue to see issues connecting to your service and do not already have a Support ticket opened, please open a support ticket for further investigation.
Affecting System - Network
During this migration, we are fortifying our 200Gig infrastructure backbone in our Dallas, TX data center location. This is a major upgrade in the quality of service we can provide you as a client.
This network upgrade and many other additional services and features are planned for the near future that we cannot wait to share with everyone.
During this maintenance period, VPC’s will need to be migrated. During this time, there may be upwards of 5 minutes of random, sporadic network drops as each VPC goes through migration. Your web applications may experience random, temporary slowness throughout the maintenance period.
Thank you for your business and for being a part of Spin Servers as we change the industry of dedicated hosting.
Affecting Other - dedispec.com
There is a scheduled maintenance window to perform upgrades on our website. During maintenance you will be unable to reach our billing/support pages and entire client facing website. Our staff will be standing by to assist you via Facebook Messenger and Skype which can be found below:
Current start time for this maintenance will be 0100 EST Sunday 03/21/2021 and end on or before 0300 EST Sunday 03/21/2021.
During this time there will be no impact to client servers.
Affecting System - Infrastructure
Updated at 1116 EST There is currently an issue with our infrastructure at the WV1 datacenter which is affecting many of our customers at this location. We have multiple staff on-site at this time to investigate all issues and resolve them as soon as possible. Please refrain from opening multiple tickets and just have 1 ticket open per server for any issues you may be seeing and our staff will continue to assist you. Our L1 staff will not have all the answers at this time and will escalate your ticket to the appropriate on-site staff who will directly resolve any issues. Thank you very much for your patience.
The majority of all affected servers are now back online. If you are still experiencing issues, we will respond to your ticket shortly as we are still working to get a small number of remaining servers online. If you do not have a support ticket open yet, please open one.
Affecting System - All WV1 Services
Hello,
We will be replacing a supervisor module in our core router tonight at 1:00AM EST, we expect a short period of downtime as we swap and reboot our core router. We are estimating 10-20 minutes of downtime starting at 1AM EST.
This will affect all services located in our West Virginia DC location (WV1)
If you are experiencing any issues after the scheduled downtime, please open a support ticket by going to https://www.dedispec.com/client/submitticket.php?step=2&deptid=1
Thank you for your patience while we complete this maintenance in order to further improve the quality of our infrastructure.
Affecting System - Network
We're currently investigating network interruptions impacting portions of the KC1 facility. This will be updated as we have updates to provide.
Thank you for your patience.
[UPDATE #1 - 8:00PM CST] - There is currently a fiber cut in the area that is impacting a portion of the KC1 facility, which is the cause of the network interruption for some service. Crews are currently working to fix the cut currently. We'll provide additional updates as they are available. A more detailed report will be sent out this evening.
[UPDATE #2 - 12:51AM CST] - We believe to have a temporary solution in place at this time. Please be aware that there may still be some connectivity issues as crews continue to complete the fiber cut repair. If your service is still offline, please either open a new ticket, or respond to your existing ticket and our technicians will look into it. Please keep in mind, ticket volume is still high but we're doing or best to get to tickets as quickly as possible.
Affecting System - Network
We're performing emergency maintenance on a switch in our Kansas City data center. This is impacting one rack and should be completed shortly.
Affecting Other - Network
We're experiencing some connectivity issues within our West Virginia location. We'll update this as we have more information to share.
Affecting Other - Cooling
We're currently experiencing cooling issues impacting a portion of the KC1 data center. Technicians are on-site and actively working on the issue. We'll update as we have more information available.
UPDATE #1: We experienced a cascading HVAC failure, forcing us to shut down a portion of the datacenter. We are still addressing this with the HVAC technicians and will provide additional information once available.
Update #2: We experienced a failure in one of our larger AC units today that caused a cascading effect causing other AC units to also fail. Due to this, a portion of the facility had to be shut down to mitigate the heat. The servers that are offline will be powered up around 10:00PM CST and we have additional staff on site to assist with this. This may take some time but we will also update users as we see their servers responding.
Thank you for your patience
Affecting Other - Cooling Infrastructure
Hello,
We're experiencing cooling issues in a small portion of the data center which has caused some services in 2-3 racks to go offline. We're working as quickly as possible to mitigate this and get any impacted services back online. If you're service is down please open a ticket so we can monitor it. Response times may be slower than normal, but we'll provide updates as we have them.
Thank you for your patience,
Dedispec, LLC.
Affecting Other - Infrastructure
We're currently investigating reports of outages in our West Virgnia data center. We'll update this as more information becomes available.
Update #1: We've been informed by our fiber provider that there is a failed switch in-between our West Virginia data center and Ashburn, VA. We haven't been given an ETA yet as their techs are working to resolve this ASAP. We'll update again as we receive more information.
Update #2: Services have been restored and should be stable at this point. Maintenance is being completed still but it shouldn't interrupt any services. If your service is still offline, or it goes back offline, please re-open your original ticket if you had one, otherwise please open a new support ticket. Thank you for your patience during this outage.
Affecting Other - Facility
We're investigating reports of outages in a section of the Kansas City facility. We'll update this as we have additional information.
Final Update:
Strong winds in the area caused a transformer up the road from the data center to pop. That impacted a single phase of the electrical system into the data center. The UPS attempted to hold the DC for multiple successive very small voltage drops and discharged, which is why a small section ended up losing power which impacted some services. The power utility company is still investigating on their side on their transformer up the road. The issue is resolved on our end and that will be the final update on this outage. If your service is still offline please open a support ticket. Total outage time was 38 minutes, most services came back online right away but some required manual intervention.
Affecting Other - n/a
We're investigating reports of outages in our West Virginia facility. We'll update this as we know more.
Affecting Other - n/a
We're currently investigating reports of servers down within the Missouri data center. We'll update this as more information becomes available.
We encountered a power related outsge to 2 rows of servers last night. The primary outage lasted about an hour but there were some services impacted longer.
UPDATE: At this time most services should be back online. If your service is still down we ask that you open a support ticket. Our engineering team is still working on a small section that is still impacted.
Affecting Other - Network Equipment Movement
Due to multiple incidents with our main network provider to the WV1 facility in the past week we are having our connection to the network provider moved temporarily to a different switch within their facility while permanent replacement hardware arrives to their facility. The maintenance window for this will be tonight from 2100 to 2130 EST as we are aiming to have this move completed within 30 minutes there is the possibility this would be decreased or increased based on possible issues that may arise during the change of switch. This will be a temporary solution which we are having completed to avoid further downtime in the coming week as the exisiting switch equipment has proven to be unrealiable to our operation. Another scheduled network maintenance will be posted once that date and time are confirmed. Thank you for your patience during this and we apologize for any inconvenience this may have caused while we work to resolve the issue.
Affecting Other - Network
We're aware of additional issues being reported from our West Virginia customers about connectivity outages. It appears to be due to another outage from our upstream provider in Ashburn. We're working with them as quickly as possible to resolve this and will update this as we have more information.
Thank you for your patience.
Affecting Other - WV1 DC
We are currently experiencing issues with our primary upstream network provider a the WV1 facility in Berkeley Springs, WV. Our network engineers are working with the provider to resolve this issue on their end as quickly as possible.
This notification will be updated throughout this outage as well as once resolved. If you have any further questions please open a support ticket or keep checking this page for updates. We apologize for any inconvenience this may cause.
Affecting Other - Power
[UPDATE - 4:16pm CST] - We've restored a large majority of our services but some still are being impacted by the power outage. We encourage you to open a ticket if your service is down so we can work on restoring power to any impacted services. The power company is still doing work on their end to resolve the issue as well. Thank you for your patience.
We're currently experiencing power issues within our Kansas City, MO facility due to one of our local power companies experiencing issues on their end. We've restored a majority of the impacted services within the facility and continue working to restore the rest of the impacted services. We will provide updated as the situation is resolved.
Thank you for your patience.
Dedispec, LLC.
We are currently experiencing some issues with our internal routing tables. Our Network Team is currently on site working to correct the problem at this time. We will update all impacted customers when we have more information on this matter and it's resolved.
Affecting System - Network
Hello,
We are currently aware of the network issues, our network engineers are currently working to resolve this issue as quickly as possible.
You will be notified as soon as we have more information regarding this issue.
Thanks
Dedispec, LLC
Affecting Other - Network
Impact: Brief disruption in traffic flows for all hosting customers, co-location customers, and downstream transit customers.
Explanation:
------------
Network engineers will perform a combined maintenance to upgrade the management module and router firmware revision on our edge border router. The new router firmware will be staged before hand on the new management module, along with the current configuration to ensure minimal disruption and downtime for all customers. Customers will notice a brief disruption to traffic flows during the swap-out of the management module, and possible transient forwarding loops during BGP re-convergence. Engineers will be following a pre-planned maintenance script to minimize impact to our customers service. We apologize for any inconvenience this maintenance may cause, and appreciate our customers understanding as we continue to build additional capacity and redundancy into our network infrastructure.
Affecting System - Core Switching Hardware
Our network engineering team will upgrading core switching hardware to double available capacity between our edge and distribution routers. This maintenance will be pre-scripted and all efforts will be made to minimize customer impact and downtime.
Maintenance Date: Monday, April 21st
Start Time: 16:00 CST
Duration: 30 minutes
Limited duration disruptions (~60s) to customer traffic flows across our core datacenter switching infrastructure.
Affecting System - Network Core Switch Chassis
Hello Dedispec Customer,
Beginning at 15:00 CST on April 15th, Dedispec network engineers will preform an emergency maintenance on one of Dedispec's core switch chassis. The maintenance will require a reload of an active management module, resulting in possible transient disruptions on all Dedispec customers traffic for a brief period (~5 minutes). Our engineers will be following a scripted / planned procedure ensuring minimal downtime for our customer base. We apologize for any inconvenience that may result from this maintenance.
Thanks,
Dedispec
Affecting System - Network
Hello,
To accommodate the rapid growth and expansion of our network infrastructure, we will be performing a planned network maintenance on Tuesday, January 21st, 2014 at 5:00pm central standard time.
A portion of this maintenance will involve software upgrades on our edge routing devices, and will require a brief service impacting outage to reload management modules. As we understand the importance our customers place on the availability of our service, we are taking all steps to minimize the duration of the impact to our customers, and expect the outage to last ~60 seconds. However, to accommodate for any unforeseen issues we have allotted a 30 minute maintenance window in the event we are forced to roll back changes or troubleshoot issues with the new router code.
We appreciate your understanding as we continue to evolve our network infrastructure and build in additional redundancy and resiliency to our service offerings.
Best regards,
Dedispec, LLC.
Affecting Other - Network
Hello Dedispec customers,
Some users may be experiencing an outage with their server at our Kansas City location. We had a Network Card go out on us and we're currently in the process of getting this swapped out ASAP.
Thank you for your patience for anyone affected.
Affecting System - Dedispec Network
Update: Connections are starting to pop up again and services should begin to start coming back up. You may notice some latency issues while we finish bringing everything back online. We'll update this again when the fiber line is fully back to normal, but currently most, if not all, services should be online again at this facility, just some possibly latency spikes.
Dedispec is currently aware of our network outage and is working quickly to resolve this. Earlier this afternoon we experienced a fiber cut. We will follow up with more details once updates are made and is resolved.
We apologize for any inconvenience and we're working quickly to resolve this. Not all customers are experiencing issues but our Midwest facility clients are.
Thanks.
Affecting System - Network
Dear Dedispec customers,
As a valued customer, Dedispec works hard to make sure you're always informed of any changes to our network that may alter your service with us. This following email is to alert you of the planned network maintenance taking place next week.
Start Time: 5/29/2013 at 5:00pm CST
End Time: 5/29/2013 at 6:00pm CST
This outage will effect all Dedispec Dedicated Server customers. Please schedule your peak times around this maintenance and make sure you inform your customers so as to minimize any issues. We expect downtime to be no longer then 20 minutes, but we have booked the entire hour to plan for unforseen issues if they occur. The purpose of this maintenance is to install new core routing equipment and to increase bandwidth capacity. We will have our network engineers all on-site to make this go as quickly as possible and to eliminate any possible problems. If you have any questions or concerns please submit a ticket or send an email to support@dedispec.com.
Once again we appreciate your patience while we go about this planned maintenance. Please send any feedback, questions or concerns that may arise. We appreciate you using Dedispec for your hosting solutions.
Thanks,
The Dedispec Team
