Skip to content

Handle errors while scaling kubernetes cluster #8107

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shwstppr merged 4 commits into from
Dec 12, 2023
+5 −6
Merged

Handle errors while scaling kubernetes cluster #8107

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
623b018
Handle errors while scaling kubernetes cluster
harikrishna-patnala Oct 17, 2023
03d1295
Fixed state issues and also applying new compute offering to all nodes
harikrishna-patnala Dec 7, 2023
6a419bc
Format code
harikrishna-patnala Dec 7, 2023
ccf9d1e
revert manual state changes
harikrishna-patnala Dec 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions ...cloud/kubernetes/cluster/actionworkers/KubernetesClusterResourceModifierActionWorker.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -767,12 +767,12 @@ protected String getKubernetesClusterNodeNamePrefix() {
return prefix;
}

protected KubernetesClusterVO updateKubernetesClusterEntry(final Long cores, final Long memory,
final Long size, final Long serviceOfferingId, final Boolean autoscaleEnabled, final Long minSize, final Long maxSize) {
protected KubernetesClusterVO updateKubernetesClusterEntry(final Long cores, final Long memory, final Long size,
final Long serviceOfferingId, final Boolean autoscaleEnabled, final Long minSize, final Long maxSize) {
return Transaction.execute(new TransactionCallback<KubernetesClusterVO>() {
@Override
public KubernetesClusterVO doInTransaction(TransactionStatus status) {
KubernetesClusterVO updatedCluster = kubernetesClusterDao.createForUpdate(kubernetesCluster.getId());
KubernetesClusterVO updatedCluster = kubernetesClusterDao.findById(kubernetesCluster.getId());
Copy link
Copy Markdown
Member Author

@harikrishna-patnala harikrishna-patnala Dec 7, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bigger change because, I observed createForUpdate() is returning an object with all null or default values and assumption is that we have to update that entry for all the columns.

This is causing few issues with scaling operations

  1. When I tried to change the node count and change the compute offering at the same time. Compute offering change on few nodes is missing. This happened because of above code
  2. Above issue is causing after effects where I could not change the compute offering of the cluster anymore as there are differences in the compute offerings of the nodes
  3. The state issue, causing NPE (the actual bug raised here)

if (cores != null) {
updatedCluster.setCores(cores);
}
Expand Down
5 changes: 2 additions & 3 deletions ...rc/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@
import org.apache.log4j.Level;

import com.cloud.dc.DataCenter;
import com.cloud.exception.ConcurrentOperationException;
import com.cloud.exception.InsufficientCapacityException;
import com.cloud.exception.ManagementServerException;
import com.cloud.exception.NetworkRuleConflictException;
Expand Down Expand Up @@ -299,8 +298,8 @@ private void scaleKubernetesClusterOffering() throws CloudRuntimeException {
boolean result = false;
try {
result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(), new HashMap<String, String>());
} catch (ResourceUnavailableException | ManagementServerException | ConcurrentOperationException | VirtualMachineMigrationException e) {
logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s", kubernetesCluster.getName(), userVM.getDisplayName()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
} catch (RuntimeException | ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException e) {
logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s", kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
}
if (!result) {
logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s", kubernetesCluster.getName(), userVM.getDisplayName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
Expand Down