• Support
  • Forums
  • Blogs
A New Community Experience is Coming! For more information, please see our announcement.

Update: Version 5.6 has been pulled from production

SkylarTalleySkylarTalley

AlienVault Employee
+12

Update, 10/30:

An updated release schedule has been posted here.

Update, 10/16:

The v5.6.1 release is still undergoing testing and optimization. The Appliance Team is committed to delivering high-quality releases and will continue to hold this update until we are confident we have sufficiently addressed a correlation performance issue that exists in v5.6. This decision was not taken lightly and we apologize for the long overdue update.

Update, 10/04:

The Appliance Team has decided to further delay the release of v5.6.1 to perform additional benchmarking and performance tests. We expect to have an updated timeline for the release early next week.

Update, 9/14:

We are now projecting to release v5.6.1 during the first week of October. Thank you for your patience.

Update, 8/23:

We have now resolved most of the support cases related to this incident. If you are experiencing any issues, please reach out to AlienVault Support directly for assistance at [email protected]

We will continue to update this forum post with information regarding 5.6.1 as it becomes available. Thank you.

Update, 8/20:

We are working aggressively to make USM Appliance 5.6.1 ready for release. We expect that this release will be available mid-September. This release will separate the update from the database migration (optional). We will continue to provide updates and more information on this thread.

If you are currently experiencing issues related to USM Appliance 5.6, please contact AlienVault Support ([email protected]) for assistance.

OSSIM users can refer to an earlier comment in this thread from Kratos for self-guidance.

Thank you for your understanding as we continue to work on this issue.


Update, 8/8:

Thank you for your understanding as we continue to work on this issue. Please be aware that we may have longer than usual wait times this week while our Support Team works to restore service for all affected customers. We have prioritized support cases related to this incident, which may impact wait times on other support cases. We appreciate your patience and cooperation with our Support Engineers, who are working diligently to get cases resolved as quickly as possible.

Update, 8/7:

We apologize for any disruption caused by this incident and are working through it as quickly as possible. Below, please find the latest update on our progress. We will continue to update this forum thread to keep you informed.

What Happened
On Thursday, August 2, 2018, we released USM Appliance 5.6 and the ability for existing USM Appliance customers to update. This update was coupled with a database migration. Because the update and the migration were coupled, the operating system of the USM Appliance could not be fully configured until the database migration was complete. Unfortunately, for some customers, that database migration took a significant amount of time. We didn't clearly state that this process could be lengthy. Customers who chose to reboot their USM Appliance before the process finished were unable to use the system.

We recognize that we did not provide enough guidance to customers ahead of the update on the potential length of time the database migration could take, how to check progress on the migration, or explicit guidance not to reboot. We will take steps to prevent this situation from happening in future updates.

What We’re Doing Now
The AlienVault Support Team has prioritized related support cases and is working with all affected customers to get their USM Appliances operating normally. Please know that we are working as quickly as possible to restore service for all affected customers.

At the same time, the AlienVault Engineering Team is working to decouple the database migration from the 5.6.1 update, which includes functional changes, bug fixes, and other benefits. This will provide customers who have not yet upgraded to 5.6 with the ability to perform the database migration separately. We will continue to provide public updates in this location.

What This Means for You
Affected customers should reach out to AlienVault Support ([email protected]) for assistance getting USM Appliance up and running. In most cases, we will be able to help you to complete the update to 5.6. For a handful of customers, we will need to help you restore a backup or redeploy. Once your USM Appliance is operating normally, you will be able to update it as usual when the 5.6.1 upgrade is available.

Customers who have successfully updated to 5.6 are unaffected and do not need to take any action. When the new 5.6.1 upgrade is available, you will be able to update it as usual. Your database was successfully migrated in the 5.6 update, so you will not need to take any action after this update. There is no risk or impact to you at this time.

Customers who did not start the 5.6 update and are still running 5.5.1 are unaffected and do not need to take any action. When the new 5.6.1 upgrade is available, you will be able to update it as usual. Separately, you will have the option to migrate your database, if you choose to.

What We Are Doing Moving Forward
We continually strive to assure the capabilities of our product and remain focused on ensuring that we deliver what you expect. Our engineering team has triaged the causes of this incident. We are implementing steps to prevent this type of situation from reoccurring.

Original Post, 8/3:

Summary
AlienVault has elected to pull the v5.6 update from production after becoming aware of several issues within our customer base.

What’s Next
We will be releasing a new version of v5.6 shortly. In the forthcoming release notes, we will detail the issues that were observed and what actions we took to address these in the new release.

What You Should Do
To those waiting for the update, we will have a new version available as soon as possible. If you have not updated to v5.6, you are not affected. Please contact AlienVault Support with any questions or concerns.

We are grateful to everyone who reported the issues and will continue to work to resolve them.

kratosAndreaPerfettianton.malakhovKyleKatAnfeldt24kaiocls

Share post:

«1

Comments

  • To be honest this is very disappointing and not acceptable for us and all of our customers to postpone an update several months and release it without proper testing! We heard it several times from AlienVault that Testing and QA will be improved but it seems that there was no improvement over the last couple of months.
    I don´t know how you have tested the release but regarding to the feedback of all the other users in the forum it seems that you have tested it without having any events in the database. Our test appliance has round about 40 million events in the database and is still updating since a couple of hours. For my point of view this should be one of the first scenarios to test it on an appliance with a usual amount of data!

    Please provide further information regarding the update on a daily base. We´re all waiting for it.


    owl06majinpowerstracy.dangernicholasalbright
  • 12 hours after starting the update and it's stuck on the database schema portion... not even the web GUI will load now due to an unspecified database error. 
    tracy.danger
  • I ran on out test systems, after more that 30 hours the database was corrupt.  Had to restore back to 5.5.1, good thing this was only a test system and not a customers production system.  Now I also did the upgrade on our test OSSIM version systems and it went fine, even with 40 million events in the database.
    tracy.danger
  • If update 5.6 has been pulled from production how is this going to impact servers still in the updating process and ones that have finished?

    We still have systems trying to update that have been running over 76 hours now. 
    tracy.danger
  • Alright, so what about those who run 5.6. now? We performed the upgrade, had (/have) our fair share of issues, but alas, we run 5.6.

    Dont tell me we're gonna have to downgrade in order to get a fixed 5.6 please.
    SHSDavid
  • Thank you all for your patience as we work to resolve this. Please see the updates added to the main post above for additional information. We will continue to update this forum thread to keep you informed.
    AndreaPerfettihbaxamoosa
  •    For any USM Appliance customers that have attempted to update their USM to the 5.6 update and has had it fail, please reach out to our Support Engineers and open a support ticket so that we can complete the update. As long as the .dpkg packages have downloaded successfully (which they should have), we can swiftly resolve the issue via a LIVE linux boot. If you have OSSIM (and/or wish to attempt the update yourself) and run in to issues, please feel free to PM me; I will answer as soon as I am able. 

      Regards,

    - kratos 
    AndreaPerfettijkisieliusBBankshbaxamoosazparkerKyleKat
  • edited August 2018
       I am waiting to hear back from @daniil to see if the procedure I've typed up is sufficiently obstacle-free. If so, I will provide the steps for resolution. I've successfully updated systems via screen-shares, but want to ensure that what I've typed up can easily be followed by users in the community. Thank you for your patience, and again, if you have AlienVault USM, please open a Support ticket so our Engineers can get you resolved. If you have OSSIM, please PM me for the procedure and let me know your results. 

       Regards,

    - kratos 
    jkisieliusRCSeczparker
  • The post above has been updated. Thanks again for your patience as we continue to work on this.
    kratosRCSec
  • edited August 2018

    You can successfully finish any balked update by LIVE booting a Linux ISO on your system. 

       First, start by downloading an x64-bit LIVE Linux image; I like Lubuntu because it is lightweight and does the job just fine :: 


       
       If this is a hardware device (bare-metal or other) you will need to write that ISO to a USB, and force the system to boot from the USB device. You can follow the procedure, below 

       * Write USB :: 

       * BIOS ::



        If this is a hypervisor (VmWare/Hyper-V, etc.), you will simply need to mount the ISO to a CD/DVD-ROM on the host and update the BIOS on that virtual machine so it boots from the CD/DVD-ROM. 


       Once you have successfully booted from either the USB or the ISO, please select the "LIVE" option from the Lubuntu menu; you do not want to install Lubuntu, but simply leverage the LIVE boot capability. Once you're LIVE booted, you open a terminal, sudo up, and find your HDD (most likely {but not always!} /dev/sda) :: 

    # sudo su 

       
       Now with #sudo privelage, we can mount the necessary directories, so you can gain access the Console Menu to USM/OSSIM, and you can then #jailbreak the appliance :: 


    1) # mount -t ext4 /dev/sda1 /mnt
    2) # mount --bind /dev /mnt/dev
    3) # mount --bind /sys /mnt/sys
    4) # mount --bind /proc /mnt/proc
    5) # chroot /mnt


       With the menu loaded, please #jailbreak the appliance. Then, issue :: 

    # dpkg --configure -a
    # sync

        If you are prompted for anything, just select 'yes' (Y). Next, install the MySQL package (this will allow the rest of the update to complete).

    # dpkg --force-install -i /var/cache/apt/archives/alienvault-mysql_10%3a5.6.0-30_all.deb


    **** 
       If for whatever reason, you do not have the 'alienvault-mysql-xxx-5.6' package, please try installing whatever version you have available. You can find out by running an 'ls' command :: 

    # ls -ltrh /var/cache/apt/archives/ | grep alienvault-mysql               
    *****


       Once completed, reboot the USM as normal (remove the ISO or USB). After the system successfully reboots, #ssh in, #jailbreak the system and run :: 

    # dpkg --configure -a
    # sync
    # apt-get update && apt-get -f install && alienvault-update -c -v -d

       
       After the system has completed the update, please reboot your system for good measure. Please let me know your endeavors. 

       Regards,

    - kratos 
    AndreaPerfettitracy.dangerRCSecjkisieliusBF
  • Is it safe to update to 5.6 now?
    or is it still touch and go?
    tracy.danger
  • Does reinstalling alienvault-mysql helped someone?  If yes, could someone share the .deb package? 
    I don't have any of them by some reason.
    Thanks.


    tracy.dangerian.ward
  • Sensor appliance died in the process of update to 5.6.  It effectively corrupted the file system and required a full rebuild.  We now have a sensor that was completely rebuilt from 5.2 media and offline updated to 5.5.0.  Now it's looking for the 5.6 update.  We can't update to 5.5.1 as a result and there is no 5.5.1 offline update available.  
    hitmanmicsnare
  • Same issue here, cannot restore the config backup because the version are different.
  • hey there,

    any update when the fix (5.6.1 // regular os-updates and general fixes) will be available?
    the support team told us that the initial 5.6.x update would be now split into two separate updates.

    first and foremost the OS update and general fixes
    and after that (followed a weeks later) the updated database fix that should improve the SIEM search performance, etc...

    it would be a cool to have a litte more communication on the proposed time-line.

    cheers
    jbhehomatracy.dangerhitmanmikeB
  • bumping this, so that Skylar sees it ;))))
    tracy.danger
  • Platinum partner and only notifcation or updates about this we get is through the forums....Nothing from account managers or blog posts?


    jbhehomahitmanmikeBnicholasalbright
  • @Ardoneye echo your statements, we are in the same boat. Maybe they are too busy re-branding everything with the AT&T logo.
    micsnaremikeBBBanksnicholasalbright
  • for anyone who hasn't seen the latest update by Skylar at the top....

    We are working aggressively to make USM Appliance 5.6.1 ready for release. We expect that this release will be available mid-September. This release will separate the update from the database migration (optional). We will continue to provide updates and more information on this thread.
  • it´s strange we are talking about just sub release 5.5->5.6 
    i will not know what happen if they want to go from 5.6 -> 6.X ;-)

    yes..it seems they fixing stuff but i can't still understand the delay of fix

    nicholasalbrighttracy.danger
  • i honestly don't think there will ever be a 6.x release.
    alienvault has announced the USM appliance as "end of sale" and soon it will become "end of support". maybe in 1-2 years, until then we will only receive security hotfixes but i can't see any major innovations or integrations to be honest.
    alienvault wants us to migrate to usm-anywhere (cloud), which has a totally different setup (backend and frontend).

    i wouldn't be surprised if usm-anywhere was indeed the previously planned v6. (6.x)

    it's a shame because i think usm-anywhere isn't bad, but I can't just migrate customer-related data into the cloud. an on-premise solution would be appreciated.
    nicholasalbrightBBanksBFian.wardtracy.danger
  • I would like to see them come up with a hybrid approach that would allow Anywhere and Appliance to work together.  Have customers that need to store their data onsite so Anywhere will not work, but could really use the cloud pieces that are in Anywhere.
    BBanksBFian.wardStefanKahntracy.danger
  • that's the point....on prem solution are very very rare at the moment.
    i can just image the vendor want cut the maintenance and support costs and of course to protect the knowledge.

    but yes i can also not imaging that ver. 6 built will be release.


  • does the latest update by skylar mean that the new update 5.6.1 (?!) has now been released for the public? when will we be able to ran the new update safely?!
    tracy.danger
  • nope...i believe they need a little bit more..
    Anfeldt24tracy.danger
  • Will 5.5.1 update to 5.6.1?
    Or so i need to update to 5.6 first? 

    Has the risk of 5.6 been mitigated?
    tracy.danger
  • 5.5.1 will directly upgrade to 5.6.1 once it is released, 5.6.0 has been pulled and not released.  The issue with 5.6 was with the database change, this will not be in 5.6.1 just the security fixes and some other improvements. The database change will come at a later time once all the bugs have been worked out.
    tracy.danger
  • Thanks hitman.
    tracy.danger
  • We installed 5.6.0. If the DB fix will not be fixed in 5.6.1, should we just scrap it and go with something else? Right now its not very useful. 
    nicholasalbrighttracy.danger
Sign In or Register to comment.