AWS migration / solved with local storage on and storing to mounted s3 space with riofs
Important Warning!
Riofs has very dangerous bugs! Dont use it!
_______________________________________________________________
Hello as I read from Alex`s explanations in the forum posts, selecting AWS as the storage location after we started the image containing modules will not work and they will continue to put to the initial local setting.
Is there a migration script to ease the transition?
If not for a site at the beginning like mine what do you think for these scenarios?
Scenario 1
If the directory structure is exactly same as local in S3
we can move the directories there. I found a mounting tool for S3 buckets that works on Debian (s3fs)
If we may move all directory structure and contents there and change the settings (?) about storage in database
Scenario 2
If the directory structure is not same as local in S3
My site doesnt has to much content yet.
I may uninstall and reinstall the modules that has images and manually create the content again
Any ideas and explanations 😊 ?
PS: AWS is continously warning about the public settings. Asking to create a user and use the system over that user. May be UNA team may explain us how to do the settings there to interact correctly with UNA scripts.
-
- · Baloo
- ·
You make me laugh, I have a doctor who was afraid to switch to digital prescriptions. Whenever I see one of your posts, I think about it and laugh. Good question, I have not played with online storage yet. When you go to master all this, I will come to you for advice :-)
-
- · Baloo
- ·
But if the structure is not the same, and if for some reason X, you stop S3, after 5 years of activity, you can not recover? it would surprise me a lot. If it is the case it is better to warn in big characters ....
-
- · Cem
-
·
In reply to Baloo
- ·
Yes Baloo :)
I can say I am not a normal medical doctor :)))
I have engineer mind :)
-
- · Cem
-
·
In reply to Baloo
- ·
That's very important but I guess Una team also thought about this.
I hope it's the same.
In any case for a regular structure it is not very difficult to write a small script to re form it.
This AWS thing is very deep. Una team should tell us how to structure the bucket configuration there with all details.
-
-
·
Alex T⚜️
- ·
There is Scenario 1 - directory structure is exactly same as local in S3.
But the problem is in public/private setting for particular file in S3, to make it work you need to upload all files as public, but in this case all private files will be public, which maybe inappropriate for some situations. So some script is needed to mark particular files as private in S3 after moving. In the future we'll try to write some tool which will help with moving files between UNA filesystems.
-
·
Alex T⚜️
-
- · Cem
-
·
In reply to Baloo
- ·
You are welcome Baloo :)
This AWS is incredible but it takes time to learn things.
I am reading now. I am a bit free in these days.
-
- · crewmark
- ·
Is there any documentation on how best to set up S3 storage?
I'm building a new site and have a SW bucket set up as "Objects can be public".
The bucket is setup with encryption.
I've filled in the keys in my Una install but have not filled in the "AWS custom domain" field.
Is this required for this to work? If so I know I can view the AWS docs at https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.htmlHowever, first assuming an AWS cusrom domain is not required then I try to upload an image it won't work and I get a S3::putObject(): [AccessDenied] Access Denied error on my server.
I do auto backups to another bucket from the same server and no problems. Does the Bucket for Una need to be set up differently?
Sort of feeling my way around in the dark here a bit
-
- · Cem
- ·
HI crewmark,
If you will not use commands on file changing like rewriting saving on files.
As I observe object based systems are good for static content. So you can serve images from there.
Also for backups it is good.
I am using riofs for about 5 months now, it is working quite good.This mounts the s3 bucket to a directory under your file system.
In this case you may leave the UNA settings as local.
-
Hi crewmark !
You may find the details about S3 on the following pages:
https://github.com/unaio/una/wiki/Dev-Storage
-
- · crewmark
-
·
In reply to Cem
- ·
Thanks Cem... I'll keep that in mind but at the moment I want to try to keep it simple :)
-
- · crewmark
-
·
In reply to LeonidS
- ·
Thanks LeonidS
I tried to cheat it by just removing everything out of the local directories then setting it to S3 but no luck so looks as if I'll be doing a reinstall over the weekend. Will post how I go.
-
- · Cem
-
·
In reply to crewmark
C
- ·
With riofs it will be simpler cause you will use your s3 bucket like a local directory.
We mount Una"s /storage directory to the bucket.
But in every scenario bucket can not be used as a block device. You can not effectively edit files on the s3. So only put and get :)
-
- · Scholar
-
·
In reply to Alex T⚜️
- ·
There is Scenario 1 - directory structure is exactly same as local in S3.
But the problem is in public/private setting for particular file in S3, to make it work you need to upload all files as public, but in this case all private files will be public, which maybe inappropriate for some situations. So some script is needed to mark particular files as private in S3 after moving. In the future we'll try to write some tool which will help with moving files between UNA filesystems.
Hi Alex T⚜️ I'm having the same structure on the cloud bucket. how can I make my local files public so that I can migrate from local storage to cloud storage?