AWS migration / solved with local storage on and storing to mounted s3 space with riofs

Important Warning!

Riofs has very dangerous bugs! Dont use it!

_______________________________________________________________

Hello as I read from Alex`s explanations in the forum posts, selecting AWS as the storage location after we started the image containing modules will not work and they will continue to put to the initial local setting.

Is there a migration script to ease the transition?
If not for a site at the beginning like mine what do you think for these scenarios?

Scenario 1

 If the directory structure is exactly same as local in S3

we can move the directories there. I found a mounting tool for S3 buckets that works on Debian (s3fs)

If we may move all directory structure and contents there and change the settings (?) about storage in database 


Scenario 2

If the directory structure is not same as local in S3 

My site doesnt has  to much content yet.

I may uninstall and reinstall the modules that has images and manually create the content again


Any ideas and explanations 😊  ?

PS: AWS is continously warning about the public settings. Asking to create a user and use the system over that user. May be UNA team may explain us how to do the settings there to interact correctly with UNA scripts.

  • 478
  • More
Replies (13)
    • You make me laugh, I have a doctor who was afraid to switch to digital prescriptions. Whenever I see one of your posts, I think about it and laugh. Good question, I have not played with online storage yet. When you go to master all this, I will come to you for advice :-)

      • But if the structure is not the same, and if for some reason X, you stop S3, after 5 years of activity, you can not recover? it would surprise me a lot. If it is the case it is better to warn in big characters ....

        • Yes Baloo :)

          I can say I am not a normal medical doctor :)))

          I have engineer mind :)

          • That's very important but I guess Una team also thought about this.

            I hope it's the same.

            In any case  for a regular structure it is not very difficult to write a small script to re form it.

            This AWS thing is very deep. Una team should tell us how to structure the bucket configuration there with all details.

            • There is Scenario 1 - directory structure is exactly same as local in S3. 

              But the problem is in public/private setting for particular file in S3, to make it work you need to upload all files as public, but in this case all private files will be public, which maybe inappropriate for some situations. So some script is needed to mark particular files as private in S3 after moving. In the future we'll try to write some tool which will help with moving files between UNA filesystems.

              • You are welcome Baloo :)

                This AWS is incredible but it takes time to learn things.

                I am reading now. I am a bit free in these days.

                • Is there any documentation on how best to set up S3 storage? 

                  I'm building a new site and have a SW bucket set up as "Objects can be public". 

                  The bucket is setup with encryption.

                  I've filled in the keys in my Una install but have not filled in the "AWS custom domain" field.
                  Is this required for this to work? If so I know I can view the AWS docs at https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html

                  However, first assuming an AWS cusrom domain is not required then I try to upload an image  it won't work and I get a S3::putObject(): [AccessDenied] Access Denied error on my server.

                  I do auto backups to another bucket from the same server and no problems. Does the Bucket for Una need to be set up differently?

                  Sort of feeling my way around in the dark here a bit

                  • HI crewmark,

                    If you will not use commands on file changing like rewriting saving on files.

                    As I observe object based systems are good for static content. So you can serve images from there.

                    Also for backups it is good.
                    I am using riofs  for  about 5 months now, it is working quite good.

                    This mounts the s3 bucket to a directory under your file system. 

                    In this case you may leave the UNA settings as local.

                      • Thanks Cem... I'll keep that in mind but at the moment I want to try to keep it simple :)

                        • Thanks LeonidS

                          I tried to cheat it by just removing everything out of the local directories then setting it to S3 but no luck so looks as if I'll be doing a reinstall over the weekend. Will post how I go.

                          • With riofs it will be simpler cause you will use your s3 bucket like a local directory.

                            We mount Una"s /storage directory to the bucket.

                            But in every scenario bucket can not be used as a block device. You can not effectively edit files on the s3. So only put and get :)

                            • There is Scenario 1 - directory structure is exactly same as local in S3. 

                              But the problem is in public/private setting for particular file in S3, to make it work you need to upload all files as public, but in this case all private files will be public, which maybe inappropriate for some situations. So some script is needed to mark particular files as private in S3 after moving. In the future we'll try to write some tool which will help with moving files between UNA filesystems.

                              Hi Alex T⚜️ I'm having the same structure on the cloud bucket. how can I make my local files public so that I can migrate from local storage to cloud storage?

                              Login or Join to comment.