After an upgrade to a new OVA image Storage Connector is unable to read and write to NFS-mounted storage

Problem description

After a Storage Connector server is upgraded to a new Open Virtual Appliance (OVA) image, the Group/User identifiers (GID/UID) of the syncp-storage user can become different to the ones used in the previous OVA version. The reason for this difference (for example, between CentOS 6 and CentOS 7 based OVAs) is an increase in the range of UID/GID reserved for dynamic allocation by the system. These GID/UID mismatches can make Storage Connectors that use NFS-mounted storage unable to read and write to this storage, because the numerical value of the UID/GID for the syncp-storage user is different from what it was on the previous (e.g. CentOS 6) server.

For example, below is a case where the Storage Connector that is configured with NFS will not be able to read and write to NFS-mounted storage after switching to a new OVA image.

Let us assume that Storage Connector is still deployed on CentOS 6 (e.g. the Storage Connector version is 2.12). 

When NFS was set up, the following command was executed to make the syncp-storage user the owner of the mounted directory:

sudo chown syncp-storage:syncp-storage /mnt/syncp

In our case, the NFS mount point was set up for the syncp-storage user with uid=498 and for the syncp-storage group with gid=499

id syncp-storage
uid=498(syncp-storage) gid=499(syncp-storage) groups=499(syncp-storage)

When switching to a new OVA version (e.g. CentOS 7 with Storage Connector 2.21), the UID and GID for the syncp-storage user become different (uid=996 and gid/groups=994):

id syncp-storage
uid=996(syncp-storage) gid=994(syncp-storage) groups=994(syncp-storage)

Because of this, when you try to reuse NFS mounted storage without re-executing the command ‘sudo chown -R syncp-storage:syncp-storage /mnt/syncp’, the new Storage Connector will not be able to write to the NFS-mounted storage.

The cause for this is that the syncp-storage user with the new GID/UID is not the owner of the Storage Connector directories and the storage mount point directory.

Often it is impractical to run this recursive change in ownership for all files and folders in the NFS mounted storage due to the number of files and folders (which could cause a degradation in quality of service and/or require significant downtime).

Problem resolution

Let us assume that on our old machine, the UID and GID for the syncp-storage user are:

id syncp-storage
uid=498(syncp-storage) gid=499(syncp-storage) groups=499(syncp-storage)

while on the new machine, they are:

id syncp-storage
uid=996(syncp-storage) gid=994(syncp-storage) groups=994(syncp-storage)

To resolve this problem :

1. Verify that the newly deployed Storage Connector is configured to use NFS-mounted storage.
   
   
2. Go to the old machine and check the UID of the syncp-storage user—it should be 498. Otherwise, note the UID and use it in place of 498 in Step 8 below.
   id syncp-storage
uid=498(syncp-storage) gid=499(syncp-storage) groups=499(syncp-storage)


3. On the old machine, check the GID of the of the syncp-storage group—it should be 499. Otherwise, note the GID and use it in place of 499 in Step 9  below.
   id syncp-storage
uid=498(syncp-storage) gid=499(syncp-storage) groups=499(syncp-storage)


4. On the new machine, check the UID of the syncp-storage user. 
   id syncp-storage
uid=996(syncp-storage) gid=994(syncp-storage) groups=994(syncp-storage)
   By default, the UID is 996, but could vary. If the UID matches the UID on the old machine, skip steps 7.a and c, and step 8.
5. On the new machine, check the GID of the syncp-storage group.
   id syncp-storage
uid=996(syncp-storage) gid=994(syncp-storage) groups=994(syncp-storage)
   By default, the GID is 994, but could vary. If the GID matches the GID on the old machine, skip steps 7.b and c, and step 9.
   
   

6. On the new machine, stop the Storage Connector service.

   sudo systemctl stop syncp-storage



7. On the new machine, verify that the UID and GID from the old machine are not already in use.
   
   Run the following command to check if a user is assigned to the UID that you need.
   getent passwd 498
   
   Run the following command to check if a user is assigned to the GID that you need.
   getent group 499
   
   If the query returns a user for any of these IDs, assign that user another UID or GID before you proceed.
   
   

   This operation might require an OVA restart. If an important system user was assigned the UID and GID that you want to free up (in this example, 498 and 499), the machine might not operate as expected after the restart, so you might need to redeploy the OVA.

   
   

8. Set the correct UID for the syncp-storage user.

   sudo usermod -u 498 syncp-storage



9. Set the correct GID for the syncp-storage group.
   

   sudo groupmod -g 499 syncp-storage


10. Get a list of all files and directories used by the Storage Connector
    find / -user syncp-storage
    
    

11. Set the correct ownership for the list of files and directories from the previous step. The list may vary depending on the Storage Connector version. Example:

    sudo chown -R syncp-storage:syncp-storage /*/*/syncp-storage/
sudo chown -R syncp-storage:syncp-storage /*/syncp-storage/
sudo chown -R syncp-storage:syncp-storage /etc/systemd/system/syncp-storage.service.d/
sudo chown syncp-storage:syncp-storage /usr/lib/systemd/system/syncp-storage.service
sudo chown syncp-storage:syncp-storage /usr/bin/checkmount.sh
sudo chown syncp-storage:syncp-storage /usr/bin/syncp-storage
sudo chown syncp-storage:syncp-storage /usr/bin/syncpstgconfig
sudo chown syncp-storage:syncp-storage /usr/bin/syncp-storage*



12. Set the correct ownership of the configuration directory containing the .secret file (if applicable).

    sudo chown -R syncp-storage:syncp-storage /mnt/config



13. Unmount the storage mount point (e.g. /mnt/syncp).

    umount /mnt/syncp



14. Remount the storage to the mount point by reloading the  /etc/fstab  file.

    mount -a



15. Start the Storage Connector service. 

    sudo systemctl start syncp-storage



This concludes the problem resolution steps.  The Storage Connector running on the new machine should now be able to read and write to the NFS-mounted storage.