2020/7/1 に SQL Database で次のような障害が発生しました。
7/1
Azure SQL Database – Japan East – Mitigated (Tracking ID CLCK-LD0)
Summary of Impact: Between 09:30 and 11:15 UTC on 01 Jul 2020, a subset of customers using SQL Database in Japan East may have experienced service connection failures or possible timeouts. Services utilizing SQL Databases may have also been impacted.
Preliminary Root Cause: We determined that instances of a gateway service responsible for handling traffic to and from some SQL Databases became unhealthy. This prevented some connections from completing as expected and caused downstream impact to services leveraging SQL Databases.
Mitigation: We performed a manual restart of the impacted gateways to mitigate the issue.
Next Steps: We will continue to investigate to establish the full root cause and prevent future occurrences. A full Root-Cause Analysis will be provided in the next 72 hours. Stay informed about Azure service issues by creating custom service health alerts: https://aka.ms/ash-videos for video tutorials and https://aka.ms/ash-alerts for how-to documentation.
最終的な RCA (Root Cause Analysis : 根本原因解析) としては、次のような内容が公開されています。
Summary of impact: Between 09:24 and 11:15 UTC on 01 Jul 2020, a subset of customers using Azure SQL Database, Azure SQL Data Warehouse/ Synapse analytics in Japan East may have experienced service connection failures or possible timeouts. Services utilizing SQL Databases may have also been impacted.
Root cause: Connections to Azure SQL Database and related data services go through a load balanced set of front-end nodes (Gateways) that provide directory lookup services and reroute the incoming connections to the intended backend-end nodes hosting the database. For scalability and zone redundancy purposes, there are multiple active SQL Gateway clusters in a region. During this incident, one of the SQL Gateway clusters became unhealthy having an intermittent impact on login availability. A specific network traffic pattern combined with a networking stack configuration on the SQL Gateway instances triggered an imbalance on the CPU processing of new connection requests. The persistence of such CPU imbalance over a long period of time caused high response latency and increased timeouts on connection requests. The error condition propagated across multiple instances of the SQL Gateway cluster in this region, sometimes causing a service restart.
Mitigation: Multiple SQL Gateway instances became healthy upon the triggered service restart. On further investigation, we were able to isolate the specific network pattern and the configuration setting that caused this incident and were able to reconfigure the traffic to prevent a recurrence.
Next Steps: We apologize for the impact to affected customers. We are continuously taking steps to improve the Microsoft Azure platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to):
- Fix the underlying issue that causes service restart when such a condition occurs.
- Improve the alerting logic and add identified telemetry to diagnose this kind of issues faster.
- Activate a newer SQL Gateway cluster in this region with a more efficient networking stack configuration that reduces the chances of hitting a processing imbalance.
障害の原因としては、「SQL Gateway クラスタ」の一つで問題が発生し、新規接続要求の CPU 処理の問題や、エラーが複数の SQL Gateway クラスターに伝搬しサービスが再起動してしまい、ログインと可用性に断続的に影響を与えたということになっています。
この障害が発生していた際には、既存の接続ついては接続が維持できていたが、新規の接続が許可されないというような状態になっていた環境もあったのではないでしょうか。
障害が発生している最中に、新規の接続を試してみたのですが、SQL Database の論理サーバー名への接続を行おうとしてもタイムアウトが発生 / Pre-Login のフェーズまでは進んだが、そこから先に進むことができず、エラーとなるというような状態となっており、新規の接続を確立することができませんでした。
障害が回復した後に、SQL Database のセッションの状態や起動タイミングを確認してみたところ、データベースについては再起動が行われておらず、いくつかのセッションについては接続が切断されず残ったままとなっていましたので、私の環境については、データベースでは問題は発生しておらず、ゲートウェイ経由での接続のみ問題が発生していたようでした。
以前、SQL Database v12 の接続のリダイレクトについて という投稿を書いたのですが、改めて、SQL Database に接続する際には、どのような動作が行われているのかをまとめてみたいと思います。
Read the rest of this entry »
