• Skip to main content
  • Skip to secondary menu
  • Skip to primary sidebar
The Blog of Jorge de la Cruz

The Blog of Jorge de la Cruz

Everything about VMware, Veeam, InfluxData, Grafana, Zimbra, etc.

  • Home
  • VMWARE
  • VEEAM
    • Veeam Content Recap 2021
    • Veeam v11a
      • Veeam Backup and Replication v11a
    • Veeam Backup for AWS
      • Veeam Backup for AWS v4
    • Veeam Backup for Azure
      • Veeam Backup for Azure v3
    • VeeamON 2021
      • Veeam Announces Support for Red Hat Enterprise Virtualization (RHEV/KVM)
      • Veeam announces enhancements for new versions of Veeam Backup for AWS v4/Azure v3/GVP v2
      • VBO v6 – Self-Service Portal and Native Integration with Azure Archive and AWS S3 Glacier
  • Grafana
    • Part I (Installing InfluxDB, Telegraf and Grafana on Ubuntu 20.04 LTS)
    • Part VIII (Monitoring Veeam using Veeam Enterprise Manager)
    • Part XII (Native Telegraf Plugin for vSphere)
    • Part XIII – Veeam Backup for Microsoft Office 365 v4
    • Part XIV – Veeam Availability Console
    • Part XV – IPMI Monitoring of our ESXi Hosts
    • Part XVI – Performance and Advanced Security of Veeam Backup for Microsoft Office 365
    • Part XVII – Showing Dashboards on Two Monitors Using Raspberry Pi 4
    • Part XIX (Monitoring Veeam with Enterprise Manager) Shell Script
    • Part XXII (Monitoring Cloudflare, include beautiful Maps)
    • Part XXIII (Monitoring WordPress with Jetpack RESTful API)
    • Part XXIV (Monitoring Veeam Backup for Microsoft Azure)
    • Part XXV (Monitoring Power Consumption)
    • Part XXVI (Monitoring Veeam Backup for Nutanix)
    • Part XXVII (Monitoring ReFS and XFS (block-cloning and reflink)
    • Part XXVIII (Monitoring HPE StoreOnce)
    • Part XXIX (Monitoring Pi-hole)
    • Part XXXI (Monitoring Unifi Protect)
    • Part XXXII (Monitoring Veeam ONE – experimental)
    • Part XXXIII (Monitoring NetApp ONTAP)
    • Part XXXIV (Monitoring Runecast)
  • Nutanix
  • ZIMBRA
  • PRTG
  • LINUX
  • MICROSOFT

VMware: Upgrade vCenter 7.0.3 to 8.0, stucks at 39% – Exporting VMware Analytics Service data – FIXED

16th November 2022 - Written in: vmware

Greetings friends, at VMware Explore Europe, vSphere 8 was finally announced as GA, with an impressive number of more than 18K downloads by then, bear in mind that Initial Availability was announced a month earlier.

As per usual, I was patiently waiting for confirmation that my backup software supported, or at least initially support for the release before I tried to install anything. The good news are that Veeam RnD and QA have performed extensive testing with version 11a P20220302 (build 11.0.1.1261 P20220302) of Veeam Backup & Replication, and they announce that this version is functional with vSphere 8.0.

I wanted to clarify that first thing I did was to upgrade my VCSA to the most up-to-date version for my 7.x deployment. But that din’t helped either.

Exporting VMware Analytics Service data – stuck at 39%

Went to the download page, started the upgrade process as per usual, then it get stucked at this stage, 39%, it is not clear what is happening, tried all the possible logs I could think of:

  • var/log/vmware/applmgmt/applmgmt.log
  • firstboot, and others

After waiting there for an hour, the process then moves to the next error, just a regular timeout I think, but most important, if you click in the button LOGS, it fails telling you that the vCenter has been shut down, so you can not retrieve anything from there.

The problem seems common and there are other people facing the same issue:

  • https://communities.vmware.com/t5/vCenter-Server-Discussions/Problem-Upgrading-from-7-0-3-to-8-0/m-p/2938147/highlight/false#M93774

Well, after days of battle with this, and many, many deployments some words from my friend Luciano struck to my mind.

Cleaning/reducing VMware Analytics Service data

It was that simple, there are a few articles on the Internet on how to do this, but didn’t made my life difficult and used an official KB, that contains a simple script:

  • https://kb.vmware.com/s/article/85116

The article mentions that some Customers had between 185K to 375k files under /var/log/vmware/analytics/prod, to take a quick look on our VCSA, we log in using SSH, and do a quick check:

ls /var/log/vmware/analytics/prod | wc -l

Geez! To my surprise, on my very small environment with 3 Hosts and less than 50 VMs I had 24246 files there, and thinking on how a migration process might take to query and move all of these files, I decided to download the script and run it.

You have the script on that official KB, but in case you are already here and want to move ahead with a copy/paste:

#!/bin/bash

# Copyright (c) 2022 VMware, Inc. All rights reserved.

# The aim of the script is to delete every file in following pattern:
#                         *VDDK.*
#
# The default period is set to 180 days, but can be set as an argument.
# Ex: ./cleaning_vddk_script.sh 90 (Delete every file older than 90 days).

DEFAULT_PERIOD=90
TARGET_DIR="/storage/log/vmware/analytics/prod"
PERIOD=${1:-$DEFAULT_PERIOD}

if [[ -d "$TARGET_DIR" ]]; then
    echo "Files before the clean :" 

    find $TARGET_DIR -name "*VDDK.*" -type f -exec wc -l {} +  |  wc -l

    echo "==="

    echo "Start cleaning VDDK files older than $PERIOD days..."

    find $TARGET_DIR -name "*VDDK.*" -type f -mtime +$PERIOD -delete
    if [ $? -eq 0 ]; then
        echo "Clean successful!"
    else
        echo "The clean was unsuccessful!"
    fi

    echo "==="

    echo "Files after the clean: "

    find $TARGET_DIR -name "*VDDK.*" -type f -exec wc -l {} +  |  wc -l
else
    echo "Directory not found!"
fi    

Then you can run it with the retention you want to keep, I went all in and just keep the files for the last week, everything else I didn’t mind. So, this is what I did after downloading/creating the file:

chmod +x cleaning_vddk_script.sh 
./cleaning_vddk_script.sh 7

And to my surprise, this was the output:

Files before the clean :
24246
===
Start cleaning VDDK files older than 7 days...
Clean successful!
===
Files after the clean: 
110

Thank you, VMware, now with just 110 files to migrate over, I thought it could be a good idea to trigger the wizard again, deploy a new VCSA, etc. The usual steps I did 16 times these last days. And then, it happened, it passed that moment in just a few minutes:

Not just that, but a few minutes after all the steps were completed, what a successful upgrade it was:

Some other Best Practices to know before upgrading

Thanks to Luciano, I wanted to share a few other best-practices to bear in mind before you trigger an upgrade of your vCenter:

  • Have an appliance Backup, snapshot, and native VCSA backup.
  • Check that all the DNS are in, plus the PTR as well. This means that you can ping all your VCSA, and ESXi Hostnames from everywhere in the VMware infrastructure, for example, with a simple dig -x IP of any component, it should resolve the FQDN:
dig -x 192.168.1.165

; <<>> DiG 9.16.27 <<>> -x 192.168.1.165
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7883
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;165.1.168.192.in-addr.arpa.    IN      PTR

;; ANSWER SECTION:
165.1.168.192.in-addr.arpa. 3600 IN     PTR     esxi-zlon-001.jorgedelacruz.es.

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Nov 16 16:54:09 UTC 2022
;; MSG SIZE  rcvd: 99
  • Expire certificates, or expired trusted root certificates. This is never good, and you should keep these in constant check. This KB I am sure it would help you. Your VCSA should look like this:

 

  • Have enough space for the upgrade, thanks Luciano. His blog with the steps here – https://www.provirtualzone.com/how-to-add-extra-space-to-vcenter-for-the-upgrade/
  • Having a local NTP is always a good idea, here are some steps to achieve this in Microsoft Windows, in Spanish but it reads easy – https://www.jorgedelacruz.es/2020/11/09/microsoft-como-crear-un-servidor-ntp-en-microsoft-windows-server-dentro-de-nuestra-infraestructura/

That should be it. I truly hope if you have ended up here, you have been able to resolve your upgrade problem and you are on your way to vSphere 8. Thanks for reading.

Filed Under: vmware Tagged With: vcenter upgrade 7 to 8 problem, vmware vmware 8, vsphere 8, vsphere 8 upgrade problem

Reader Interactions

Comments

  1. bala says

    2nd May 2023 at 1:09 pm

    had the exact same error , thanks for the fix , should i just restart the upgrade from stage 1 ?

  2. Robbyrob says

    11th December 2023 at 3:58 pm

    FYI this can be run WHILE the upgrade is in progress and stuck at this step!

    Thank you so much for your help

  3. Levi says

    30th January 2024 at 3:53 am

    Thank you so much for this article! I ran the script while the migration was stuck (for the 3rd time!) and as soon as it finished the progress bar started moving again and finished successfully! What a relief!

  4. jorgeuk says

    1st February 2024 at 10:54 am

    Great news!!! Awesome it worked.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Primary Sidebar

  • E-mail
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • YouTube

Posts Calendar

November 2022
M T W T F S S
 123456
78910111213
14151617181920
21222324252627
282930  
« Oct   Feb »

Disclaimer

All opinions expressed on this site are my own and do not represent the opinions of any company I have worked with, am working with, or will be working with.

Copyright © 2025 · The Blog of Jorge de la Cruz