Can anyone help me with some basic Amazon EC2 / S3 / Glacier questions?

[fuzzybabybunny]

Especially EC2. I've got a website hosted on EC2 and I'm finding it incredibly confusing to figure out what's going on.

I'm used to just purchasing hosting that has cPanel already installed and I can FTP directly into the server.

EC2 changes everything and I literally can't make heads or tails out of it.

 

[tfinch2]

So what are your questions?

 

[fuzzybabybunny]

So what are your questions?

Guh, I don't really know where to start since I can't even figure out the foundation of Amazon's services - and they require me to pay for tech support...

I get lost easily because explanations have a ton of jargon and nomenclature that I don't understand. I'd like something in layman's terms.

1. Keys

How does a private key or a public key or key pairs work? What are these things? I'm used to just having a frickin' master password and then I can create sub-accounts and assign passwords to those sub-accounts. Easy to understand and really straightforward. Amazon doesn't use the "password" nomenclature.

For example, I have my master cPanel password which I don't give out to anyone else. I can create custom FTP accounts in cPanel and restrict access to only certain directories and assign them passwords. If a developer quits, I can delete their FTP account or change their password for my own security.

With Amazon, I have no idea how to even give a developer access to my server files, so I just give them my master key, which I know I shouldn't do, but I can't figure out how to set up a sub-account for them.

2. I can easily create a complete backup in a .tar file of my entire website through cPanel. I can't even figure out how to create a backup of my EC2.

3. I code using DreamWeaver. With my non-Amazon site it's easy - I can manage files and get/put files all within the FTP client within Dreamweaver - cPanel gives me an FTP address in which to easily access all of the files on my server. How do I do this with EC2? How do I access my EC2 files on DreamWeaver and get/put files like I do normally?

 

[Fallen Kell]

1) Keys are for encryption. You need to have encryption of your data otherwise anyone could read/access it. PKI (Public Key Infrastructure) is pretty well standard at this point. There are 2 keys involved. When you create your encryption keys, it creates a public key, which you register, and you save the private key (NEVER GIVE THIS AWAY). Anyone who then wants to communicate with you or setup data will encrypt the data using the public key you registered. Your private key will decrypt that data.

So in this case, they never actually have access to read your data, only send you data which you can access.

Remember, in the cloud, your data may get left behind on a system (in temp, or memory, etc), and since you are not the only one who runs and uses that hardware, someone who uses that particular machine after your instance is turned off may have access to some of your data. This is why everything needs to be encrypted (and why their framework requires encryption keys).

Personally, I wouldn't trust it, but that is me...

2) Not sure (someone else will need to answer).

3) I am pretty sure they do not use FTP (FTP uses plain text, and thus both your username/password as well as the data is open for anyone to be able to snoop and access). Again, someone else may know more.

 

[uclabachelor]

2. I can easily create a complete backup in a .tar file of my entire website through cPanel. I can't even figure out how to create a backup of my EC2.

3. I code using DreamWeaver. With my non-Amazon site it's easy - I can manage files and get/put files all within the FTP client within Dreamweaver - cPanel gives me an FTP address in which to easily access all of the files on my server. How do I do this with EC2? How do I access my EC2 files on DreamWeaver and get/put files like I do normally?

2. EC2 simply provides you a "cloud" computer that you can telnet in or remote desktop into. It is up to you to install or setup extra software or do whatever you want.

3. You ftp into the EC2 instance by using your private key file in #1.

 

[Fallen Kell]

Oh, a follow up on the "Personally, I wouldn't trust it" part. The reason I don't trust it is the fact that once your data is outside your control, it is just that, outside your control. You are entirely reliant on the encryption you used to protect it until the point in time that the data wouldn't harm you, your company, or anyone else if it becomes known. For something like that, you are then relying on the encryption method used (the bit length of the key you used, the randomness of the random number generator, the algorithm used, etc.) doesn't have some flaw which can be exploited to then allow for the easy brute force attack of your key. And at this point in time, with just about every encryption algorithm currently available having been in part designed, or modified by certain government agencies, you can not trust that there is not some flaw that can be exploited. If it is something that will destroy your business if a competitor gains access to, I would not put it on "the cloud".

 

[DaveSimmons]

For EC2 servers where you store files in the instance instead of in a S3 bucket, you upload files using keys as the user-password and Secure FTP as the upload tool. I've done this with PuTTY (free tool).

For backups, you SSH (secure shell) in to the server, or write web script pages (PHP, whatever) that execute backup scripts, or add cron jobs that run backup scripts. PuTTY can do the SSH access too.

I'd suggest some Google searches for things like "Amazon EC2 DreamWeaverr", "Amazon EC2 SSH", "Amazon EC2 SFTP"

 

[tfinch2]

One question I have is if you don't know what EC2 is, why the heck are you using it?

EC2 is more of what is called infrastructure as a service. You get some chunk of resources, and you can do whatever you want to with it.

This is nice where you want to be in control of your software stack, but not so nice if you have no idea what your doing. Are you going to be doing updates and security patches? What kind of monitoring are you going to set up?

If you're just trying to run a website, why don't you look into just using a regular web host or use Amazon's platform as a service? Let someone else deal with keeping the infrastructure up to date and running.

 

[Markbnj]

With Amazon, I have no idea how to even give a developer access to my server files, so I just give them my master key, which I know I shouldn't do, but I can't figure out how to set up a sub-account for them.

We run all our stuff on EC2, and the first thing I would say is that it is not intended for use by non-IT staff. I'm not surprised that you are finding it confusing.

On the authentication front, what you want is IAM (Integrated Access Management). You'll see it under "Deployment & Management" in the AWS console home page. Using IAM you can create sub-accounts for your developers, and whoever else needs access. You can generate credentials/keys for them, and control which resources they are allowed to use/manage. However be warned that moving past the default options for account types gets complicated very quickly, since you're writing custom policy statements at that point.

2. I can easily create a complete backup in a .tar file of my entire website through cPanel. I can't even figure out how to create a backup of my EC2.

This depends somewhat on how you set your instance up and where the file system is. EC2 instances can be running on "ephemeral" storage, an EBS volume, or some combination of the two. Ephemeral storage disappears when the instance is shut down. EBS storage is persistent even when the instance is shut down, and you pretty much don't have to worry about it going away. The trade-off is that you pay a small monthly charge. If you're running on ephemeral storage then you should read up on creating snapshots or "Amazon Machine Images." An AMI is an image of your entire system that you can launch an instance from. You can do the same thing with a snapshot, for the most part.

3. I code using DreamWeaver. With my non-Amazon site it's easy - I can manage files and get/put files all within the FTP client within Dreamweaver - cPanel gives me an FTP address in which to easily access all of the files on my server. How do I do this with EC2? How do I access my EC2 files on DreamWeaver and get/put files like I do normally?

Amazon uses SSH to provide access to running instances, as well as their built-in console. To SSH into an EC2 instance you must generate a keypair, and install the private key on your local system. On Windows I use PuTTY for this, but most often these days I work from Debian. Once you have SSH working you can use SCP to manage files over it. For Windows I use WinSCP. It's as easy as FTP once it is set up.

The whole key pair thing, though, will be damned confusing. Fortunately Amazon has stellar tutorial/fast-start pages on all these topics and many more. They'll walk you right through it with screenshots and all. So if you're intent on running on EC2 and managing it yourself you should spend some time on their site and read up.

 

[fuzzybabybunny]

1) Keys are for encryption. You need to have encryption of your data otherwise anyone could read/access it. PKI (Public Key Infrastructure) is pretty well standard at this point. There are 2 keys involved. When you create your encryption keys, it creates a public key, which you register, and you save the private key (NEVER GIVE THIS AWAY). Anyone who then wants to communicate with you or setup data will encrypt the data using the public key you registered. Your private key will decrypt that data.

So in this case, they never actually have access to read your data, only send you data which you can access.

Remember, in the cloud, your data may get left behind on a system (in temp, or memory, etc), and since you are not the only one who runs and uses that hardware, someone who uses that particular machine after your instance is turned off may have access to some of your data. This is why everything needs to be encrypted (and why their framework requires encryption keys).

Personally, I wouldn't trust it, but that is me...

2) Not sure (someone else will need to answer).

3) I am pretty sure they do not use FTP (FTP uses plain text, and thus both your username/password as well as the data is open for anyone to be able to snoop and access). Again, someone else may know more.

Ohhhh... I see. I think. It protects data transfers. If data goes from point A to point B but it gets plucked mid-stream via a hacker, it is unintelligible because a private key is required to decode it into something understandable. From How Stuff Works:

"The sending computer encrypts the document with a symmetric key, then encrypts the symmetric key with the public key of the receiving computer. The receiving computer uses its private key to decode the symmetric key. It then uses the symmetric key to decode the document."

So in the case of FTP, FTP sends all the data via plain text, so people can "steal" content mid-stream while data is being transferred, right?

What I don't understand is that key pairs seem to be great for encrypting data *transfers* but why are they also used for account access?

Why can't I use just a regular password to access my server and my account, and just use key pairs for all data transfers? I feel like I have to give people a key just to *log into* the server...

2. EC2 simply provides you a "cloud" computer that you can telnet in or remote desktop into. It is up to you to install or setup extra software or do whatever you want.

3. You ftp into the EC2 instance by using your private key file in #1.

Hmmm... ok, so EC2 is basically a computer with CPU, RAM, and internet connection but with a blank hard drive that I have to load stuff into. It doesn't come with anything. No FTP client. No cPanel. No MySQL. Nothing.

For EC2 servers where you store files in the instance instead of in a S3 bucket, you upload files using keys as the user-password and Secure FTP as the upload tool. I've done this with PuTTY (free tool).

For backups, you SSH (secure shell) in to the server, or write web script pages (PHP, whatever) that execute backup scripts, or add cron jobs that run backup scripts. PuTTY can do the SSH access too.

I'd suggest some Google searches for things like "Amazon EC2 DreamWeaverr", "Amazon EC2 SSH", "Amazon EC2 SFTP"

This is exactly my problem. cPanel is great because it is a user interface that I can just click through to accomplish tasks. With EC2 I have to actually write my own scripts to run things. I have ZERO idea how to do this.

I guess I have to look into installing cPanel onto my EC2 first? And maybe like an FTP client that runs on the EC2 server? Uhhh... something like that?

 

[fuzzybabybunny]

One question I have is if you don't know what EC2 is, why the heck are you using it?

EC2 is more of what is called infrastructure as a service. You get some chunk of resources, and you can do whatever you want to with it.

This is nice where you want to be in control of your software stack, but not so nice if you have no idea what your doing. Are you going to be doing updates and security patches? What kind of monitoring are you going to set up?

If you're just trying to run a website, why don't you look into just using a regular web host or use Amazon's platform as a service? Let someone else deal with keeping the infrastructure up to date and running.

Well, I initially didn't know what EC2 was that vanilla when I signed up for it. All I knew is that the website I wanted to create is a very complex one and will have massive amounts of data, bandwidth, and heavy CPU requirements. I didn't think that a regular host would be able to supply this affordably. So I guess I figured that EC2 would be good as something to grow into, since it advertises itself as a service that only charges you for what you use, whereas most web hosts are simply tiered.

We run all our stuff on EC2, and the first thing I would say is that it is not intended for use by non-IT staff. I'm not surprised that you are finding it confusing.

On the authentication front, what you want is IAM (Integrated Access Management). You'll see it under "Deployment & Management" in the AWS console home page. Using IAM you can create sub-accounts for your developers, and whoever else needs access. You can generate credentials/keys for them, and control which resources they are allowed to use/manage. However be warned that moving past the default options for account types gets complicated very quickly, since you're writing custom policy statements at that point.

This depends somewhat on how you set your instance up and where the file system is. EC2 instances can be running on "ephemeral" storage, an EBS volume, or some combination of the two. Ephemeral storage disappears when the instance is shut down. EBS storage is persistent even when the instance is shut down, and you pretty much don't have to worry about it going away. The trade-off is that you pay a small monthly charge. If you're running on ephemeral storage then you should read up on creating snapshots or "Amazon Machine Images." An AMI is an image of your entire system that you can launch an instance from. You can do the same thing with a snapshot, for the most part.

Amazon uses SSH to provide access to running instances, as well as their built-in console. To SSH into an EC2 instance you must generate a keypair, and install the private key on your local system. On Windows I use PuTTY for this, but most often these days I work from Debian. Once you have SSH working you can use SCP to manage files over it. For Windows I use WinSCP. It's as easy as FTP once it is set up.

The whole key pair thing, though, will be damned confusing. Fortunately Amazon has stellar tutorial/fast-start pages on all these topics and many more. They'll walk you right through it with screenshots and all. So if you're intent on running on EC2 and managing it yourself you should spend some time on their site and read up.

I think I should be able to figure out the key pair stuff now, but the stuff that I don't have confidence in is using PuTTY and writing scripts. I know my way around a GUI, but that's the problem. PuTTY and scripts are not GUIs. Even if I get those up and running I won't know how to get anything accomplished on the server. I would need someone to install a GUI for me in which I can run functions on the server. And install FTP so that I have a way of accessing and editing the files of my website.

 

[Leros]

So... why are you using EC2 instead of a more managed provider?

 

[Markbnj]

I think I should be able to figure out the key pair stuff now, but the stuff that I don't have confidence in is using PuTTY and writing scripts. I know my way around a GUI, but that's the problem. PuTTY and scripts are not GUIs. Even if I get those up and running I won't know how to get anything accomplished on the server. I would need someone to install a GUI for me in which I can run functions on the server. And install FTP so that I have a way of accessing and editing the files of my website.

PuTTY actually is a GUI program made specifically for Windows. It provides an SSH shell as well as other advanced capabilities you won't need for the moment. The only oddness is that it uses it's own key format, so once you create the .pem key file and dl it from AWS you need to run PuttyGen to convert it to PuTTY's .ppk format before you can set it as the key for a connection. Again, Amazon has tutorials that walk you through this workflow specifically using PuTTY.

Once you have a PuTTY key you can install and use WinSCP, which relies on the same SSH core that PuTTY uses. That is also a GUI application that works essentially like any other file manager on Windows.

Personally, I would reconsider EC2. As Leros suggests, a managed provider will set you up small and support you through your growth, and rather than learning about how to set up an EC2 instance and shell into it you can be growing your business. I've had a very good experience with Rochen Hosting. They have awesome system performance and support, and as you grow they will be able to scale you up. There are certainly lots of other choices too.

 

[DaveSimmons]

Hosted does realistically limit you to a single server though. If FBB expects to need the instant-on multiple web server scaling of cloud services then maybe it makes sense to build it that way from the start.

FBB, note that depending on what you want to do you could also go with a hybrid model -- a hosted server for your website and SQL databases, but S3 storage buckets and possibly cloudfront for serving high traffic of bulk files.

But maybe you need to be paying someone to do a system design for you and tell you what makes sense, based on all of the secret plans you need to hide from us

 

[tfinch2]

Hosted does realistically limit you to a single server though. If FBB expects to need the instant-on multiple web server scaling of cloud services then maybe it makes sense to build it that way from the start.

FBB, note that depending on what you want to do you could also go with a hybrid model -- a hosted server for your website and SQL databases, but S3 storage buckets and possibly cloudfront for serving high traffic of bulk files.

But maybe you need to be paying someone to do a system design for you and tell you what makes sense, based on all of the secret plans you need to hide from us

Hosting does not limit you to a single server if you are using a PaaS like Heroku or one of the many other choices out there.

 

[Markbnj]

Hosted does realistically limit you to a single server though. If FBB expects to need the instant-on multiple web server scaling of cloud services then maybe it makes sense to build it that way from the start.

FBB, note that depending on what you want to do you could also go with a hybrid model -- a hosted server for your website and SQL databases, but S3 storage buckets and possibly cloudfront for serving high traffic of bulk files.

But maybe you need to be paying someone to do a system design for you and tell you what makes sense, based on all of the secret plans you need to hide from us

I agree to a point. A typical hosting service will happily scale you up across multiple servers if that becomes necessary, and might even help put in place some custom infrastructure for a price. But they won't be set up to deliver the flexibility that EC2 does. You can create auto-scaling groups for web servers, handle SSL negotiation at the load balancer, deploy additional cache nodes with a click, spread the servers in various tiers across different physical availability zones, etc. It's all very cool, but as you suggest, FBB probably would need to hire someone to design/deploy the system in order to take real advantage of what AWS offers.

 

[DaveSimmons]

Hosting does not limit you to a single server if you are using a PaaS like Heroku or one of the many other choices out there.

Sorry I didn't go into enough detail.

By hosting I meant typical "web hosting" of sites, VPS and dedicated servers like FBB is familiar with now.

Some companies offer scaling up of a sort with them, but their fees for load balancing and clusters are typically very high and are fixed monthly costs instead of based on variable demand. If you need to support a peak of 5 servers, you need to pay for that peak capacity 365 days a year.

So from a practical standpoint (aka realistically) if you need to scale beyond one server then you're better off with elastic cloud services like Amazon's.

And you're better off paying someone who knows their stuff to come up with a scalable, fault-tolerant design. For example FBB probably hasn't looked at how to set up health monitoring and how to kill instances and spawn new ones as needed, or how to use database server instances separate from the web / application server, or a dozen+ other aspects of a scalable solution.

 

[Leros]

So from a practical standpoint (aka realistically) if you need to scale beyond one server then you're better off with elastic cloud services like Amazon's.

Not sure I'd agree with that. EC2 is a lot more expensive than non-elastic VPS providers. The advantage of EC2 is obviously that it's elastic. If you need a static number of a servers, I think you'd generally be better off with a standard VPS provider.

 

[DaveSimmons]

OK, "possibly only if you need to scale above one server with a variable level of demand" then

Amazon does offer features you won't get with typical dedicated hosting though, like separate static file storage (and cloudfront CDN), separate database instances, server images that you create and can "spin up" yourself as needed (more easily than doing backup / restore to a VPS).

 

[Markbnj]

Not sure I'd agree with that. EC2 is a lot more expensive than non-elastic VPS providers. The advantage of EC2 is obviously that it's elastic. If you need a static number of a servers, I think you'd generally be better off with a standard VPS provider.

I've heard people say this, but I haven't seen a detailed cost breakdown. I know recently an automated testing company announced they were moving to a private cloud, but they spin up a lot of servers for short test runs, and the minimum one hour billing was hurting them.

I have an M1 large that I am using for some prototyping, and if I leave it running for a solid month it costs about $130. That's with.... 10GB I think of EBS persistent storage. But the upside is that I can run my stuff, then shut the instance down and billing stops.

 

[Leros]

I've heard people say this, but I haven't seen a detailed cost breakdown. I know recently an automated testing company announced they were moving to a private cloud, but they spin up a lot of servers for short test runs, and the minimum one hour billing was hurting them.

I have an M1 large that I am using for some prototyping, and if I leave it running for a solid month it costs about $130. That's with.... 10GB I think of EBS persistent storage. But the upside is that I can run my stuff, then shut the instance down and billing stops.

My company did some analysis (that I don't know the details of) and we ended up getting dedicated servers for half of the price of what it would cost to get similar capacity on EC2. These dedicated servers are in a managed facility, so we're leasing them and don't have to maintain them ourselves. We can get a dedicated server setup in a few hours after the ticket is put in, so we can still add capacity pretty quickly if we need. Due to some weekly cycles, we do have some situations where we need lots of extra capacity for several hours a week and we use EC2 to temporarily provide that extra capacity.

I use DigitalOcean for my personal server. To get some numbers into this conversation, I'll compare it to Amazon EC2.

-----

Digital Oceans's smallest plan
0.5GB RAM, 1 core, 20GB SSD
$5/mo

The comparable EC2 plan seems to be the t1.micro.
0.6GB RAM, 1 core, no local storage
$15/mo on demand or $10/mo reserved for 1 year

-----

A larger Digital Ocean plan
8GB RAM, 4 cores, 80GB SSD
$80/mo

The comparable EC2 plan seems to be the c3.xlarge
7GB RAM, 4 cores, 80GB SSD
$220/mo on demand or $164/mo reserved for 1 year

----

For an always-on server, like a webserver, something like Digital Ocean is going to be lot more affordable than EC2.

All that being said. EC2 is great for running short jobs. I'll sometimes spin up an AWS instance for a few hours to run some sort of job that would otherwise saturate my computer. Might cost me $5 for that privilege though. I've also used their Elastic MapReduce for when I need to run a map reduce job for a personal project (sadly, I don't have my own Hadoop cluster at home).