Header Image
Active Incident
Welcome to Node4 status page detailing information about incidents and planned maintenance. Please use the subscribe button top right to subscribe to this feed to be able to receive updates.

Incident Status

Degraded Performance

Components

Connectivity, Colocation

Services

Ethernet Services, Internet Transit (N4-Transit), MPLS Services, Derby DC2 H1, Northampton DC4 H1



May 20, 2018 7:58PM BST
May 20, 2018 6:58PM UTC
[Investigating] Services Affected: - CIB and MPLS. Nature of Issue: - Monitoring has detected a 10Gb link down between Derby and Northampton. We've raised the fault with the supplier and are awaiting feedback. Characteristics of Issue: - Some packet loss may have been observed briefly but we operate a redundant topology and the core will have re-converged. Next Status Update: - 20/05/18 at 22:00

May 20, 2018 10:24PM BST
May 20, 2018 9:24PM UTC
[Identified] Supplier is looking to arrange an engineer once remote checks have been made. Next Status Update:- 21/05/18 at 09:00

Incident Status

Operational

Components

Cloud

Services

N4Cloud Northampton



May 9, 2018 11:24AM BST
May 9, 2018 10:24AM UTC
[Investigating] Monitoring has alerted us to an issue with the Northampton Cloud we are currently investigating the cause.

May 9, 2018 12:32PM BST
May 9, 2018 11:32AM UTC
[Identified] One of the leaf switches has crashed due to a known bug. We are liaising with vendor TAC. We also have replacement hardware on route to site as a contingency. All traffic has failed over to resilient parts of the network

May 9, 2018 1:29PM BST
May 9, 2018 12:29PM UTC
[Identified] An engineer is currently onsite re-imaging the failed switch. If this fails to recover the device it will be replaced

May 9, 2018 1:41PM BST
May 9, 2018 12:41PM UTC
[Monitoring] Users should not currently be seeing issues but the platform remains at risk

May 9, 2018 3:04PM BST
May 9, 2018 2:04PM UTC
[Monitoring] The hard drive of the switch has failed meaning a rebuild is not possible. The replacement switch is being prepared and will be added to the live network as part of an emergency maintenance at 6pm today

May 9, 2018 6:04PM BST
May 9, 2018 5:04PM UTC
[Monitoring] Engineers are now replacing the faulty switch. Services should be considered at risk. A further update will be provided once complete.

May 9, 2018 6:52PM BST
May 9, 2018 5:52PM UTC
[Monitoring] The work to replace the failed switch may have caused service disruption, engineers are continuing to work on the issue a further update will be provided in 30 minutes

May 9, 2018 7:17PM BST
May 9, 2018 6:17PM UTC
[Monitoring] The new switch is now up and configured, there are some residual errors which are under investigation. Some customers may still be seeing service disruption

May 9, 2018 7:32PM BST
May 9, 2018 6:32PM UTC
[Monitoring] The replacement switch has been removed from the live platform following further analysis of the errors the maintenance has been backed out. Customers should see no further disruption to connectivity. We are consulting further with vendor TAC to rectify the issue

May 10, 2018 9:41AM BST
May 10, 2018 8:41AM UTC
[Investigating] The replacement of the leaf switch in last nights emergency maintenance failed. Engineers are continuing to work closely with TAC to obtain a root cause and fix as soon as possible. Apologies for any inconvenience.

May 10, 2018 9:58AM BST
May 10, 2018 8:58AM UTC
[Monitoring] The platform remains at risk this morning and is being monitored closely. We are continuing to engage with vendor TAC to add the replacement hardware back to the network. A further update will be made available at 12pm

May 10, 2018 12:22PM BST
May 10, 2018 11:22AM UTC
[Monitoring] Troubleshooting is continuing with the vendor. It is possible that a further maintenance window is required later this evening which may be service impacting. Further details to follow by 2pm

May 10, 2018 2:22PM BST
May 10, 2018 1:22PM UTC
[Monitoring] Vendor TAC have advised that they require us to perform live trouble shooting of the affected device to identify the root cause. To this end we have planned an emergency maintenance window commencing at 11pm and ending at 2am this evening. It is anticipated that during this live troubleshooting there will be periods of instability to services connected to this platform Further updates will be provided following the maintenance window tonight

May 10, 2018 11:53PM BST
May 10, 2018 10:53PM UTC
[Monitoring] Engineers have begun the phase of testing which is potentially intrusive. All testing completed so far has been completed without impact to services. The next update will be available by 1am

May 11, 2018 1:05AM BST
May 11, 2018 12:05AM UTC
[Monitoring] We have added the new leaf switch back into the network but are seeing some services impacted. This is being investigated by the engineers and vendor TAC A further update will be provided by 2am

May 11, 2018 2:12AM BST
May 11, 2018 1:12AM UTC
[Monitoring] We are continuing to investigate issues within the platform with vendor TAC although some services have been restored several services are still at risk. We are continuing to work to restore resilience to the remaining services We will provide a further update by 3am

May 11, 2018 3:02AM BST
May 11, 2018 2:02AM UTC
[Monitoring] The engineers have now patched all cables into the replacement switch . We are in the final stages of checking all infrastructure and are monitoring closely. A further update will be supplied by 4am

May 11, 2018 4:17AM BST
May 11, 2018 3:17AM UTC
[Monitoring] All services have been restored to full resiliency as of 3:30am. The platform remains under close monitoring for the next 24 hours. An RFO will be issued in due course We apologise for the inconvenience caused

Connectivity




Degraded Performance

Points of presence




Operational

Cloud




Operational

Communication




Operational

Colocation




Degraded Performance

Scheduled Maintenance

Schedule

May 16, 2018 10:00PM - May 17, 2018 2:00AM BST
May 16, 2018 9:00PM - May 17, 2018 1:00AM UTC

Components

Connectivity

Services

N4-Protect (NGFW)

Description

On Wednesday16th May, engineers will be performing maintenance on the N4Protect analytics and reporting platform. This is required as part of ongoing service improvements. Analytics and reporting systems will be unavailable as part of this maintenance. Minor configuration changes may be required on the N4Protect firewalls to accommodate the changes, but this is not expected to affect the platform. During this maintenance some logs will not be stored and therefore will be unavailable for future analysis and reporting. Impact At Risk – Wednesday 16th May 22:00 – Thursday 17th May 02:00 BST No impact is expected to the N4Protect firewalls, however they should be considered “At Risk” for the duration of the window. N4Protect logging and analytics systems will not available during this maintenance. Some logs send from the firewalls will not be stored on the analytics systems during the maintenance. Updates will be posted to N4Status during this planned maintenance. Support & Feedback Our network operations engineers will closely monitor the work and will do everything possible to minimise any inconvenience to you. If you have any problems with your connection after this time, or if you have any questions regarding the maintenance at any point, please call Technical Support at 0845 123 2229 or visit www.node4.co.uk/support. While it is important for Node4 to perform maintenance such as this to ensure the quality of our network, we do everything possible to minimise any inconvenience to our customers. We appreciate your patience and welcome any feedback. Regards, Node4 Technical Support

Schedule

May 23, 2018 12:00AM - 4:30AM BST
May 22, 2018 11:00PM - 3:30AM UTC

Components

Connectivity, Cloud

Services

Internet Transit (N4-Transit), MPLS Services, N4Cloud Northampton

Description

On Wednesday 23rd May 2018 @ 00:00 Node4 Engineers will be carrying out work to upgrade the firmware on one of of the core switches upstream to the cloud platform at Northampton We expect failover to be around 1-3 minutes or less while we re-route traffic around the core switch, you may experience slight degradation across MPLS and Internet transit paths while traffic is re-routed. We therefore deem this window AT RISK until Thursday 24th May 2018 @ 04:30

Schedule

July 1, 2018 9:00AM - 9:00AM BST
July 1, 2018 8:00AM - 8:00AM UTC

Components

Communication

Services

SIPLink Derby, SIPLink Leeds

Description

This notice is intended for Node4 SIPLink customers who use Node4's STUN service * What is STUN? STUN ("Simple Traversal of UDP over NAT") is a protocol that aims to resolve the issues that some Customers experience when using Network Address Translation (NAT) with their SIP endpoints. This is defined by the IETF in RFC3489 and RFC5389. * What service does Node4 Supply? Node4 host several STUN servers. These are public facing servers and can effectively be used by anyone - not just Node4 customers. They do not have any connections to the Node4 SIPLink platform or any other back-end systems. Current servers include: stun.node4.co.uk stun1.node4.co.uk stun2.node4.co.uk STUN does not form part of the SIPLink Service Schedule or contract and has been provided as a free of charge add on service. It does not carry any service guarantee or SLA. * Why are Node4 retiring this service? Usage of STUN by Node4 SIPLink customers is limited and is continually decreasing. This is mainly due to improvements in ALGs and the fact that most PBXs are now able to handle NAT in a more sophisticated manner. In addition, STUN is not provider dependent which means that customers do not need to use a Node4-provided service. There are many publicly available services that can be used instead, should customers still require access to STUN. In most case, where a PBX Vendor requires STUN, they will offer their own publicly-accessible solution. * When will this happen? All STUN servers will be disconnected 1st July 2018 * What if I depend on STUN? We would recommend discussing with your PBX vendor as they may have an alternative solution for NAT traversal, or may have their own STUN server available. If you have any concerns or questions relating to this, please log a support ticket via the Node4 Service Desk and we will endeavor to assist you in finding a solution.