SocialStack Cloud

SocialStack Cloud is a service for simplifying the process of deploying and managing a cluster of servers running a dotnet project. It can create servers on a variety of cloud platforms (AWS, DigitalOcean, Azure) and then setup routing including DNS and loadbalancers. This is always in the form of "bring your own cloud platform account" to ensure that you have maximum access to your own sites and services.

Currently SocialStack Cloud is a non-public service and requires an administrator account to use. As SocialStack Cloud is itself a SocialStack site, anyone who is already an administrator can grant access to a new user account by changing its role once they have registered here.

Creating a project

To create a new project, first go to the project list and click the new project button. You'll need to give your project a name and specify the cloud platform account to use. Cloud platform config is optional if you are only importing connection details for existing servers. You can also specify multiple platforms simultaneously which can be useful for large multicloud clusters. Here's sample platform configuration for the 3 main cloud platforms:

AWS

Create an access key following the guide here.

{
    "AWS": {
        "AccessKeyId": "abcabc..",
        "SecretAccessKey": "abcabc.."
    }
}

DigitalOcean

Create an access token for the account you would like to use. It's recommended to create a team, optionally per project, so you can better monitor individual project resources and their costs.

{
    "DigitalOcean": {
        "ApiKey": "abcabcabcabc.."
    }
}

Azure

Create an Azure active directory service principal following the guide here.

{
    "Azure": {
        "TenantId": "...",
        "ClientId": "...",
        "ClientSecret": "...",
        "SubscriptionId": "..."
    }
}

Cloud Build

Optionally during project creation, specify where your source is. Currently it must be a git repository and one which is accessible via classic git authentication (A personal access token over https rather than an SSH key).

Cloud build works by using the stage server of the project. This way the code of a given project is always isolated into a dedicated cluster of servers and it can do a pull rather than a full checkout for often large gains in build speed. Because of this, there are a few important notes:

• The stage server needs enough RAM to run your site and the build. The "medium" server option is usually appropriate. This is usually a battle between Babel (the javascript transpiler) which uses some 600MB on its own and MySQL which will take as much RAM as it can get. If a build is running very slowly or the stage site appears to have gone offline whilst a build is happening, it's because the server has ran out of RAM. Connecting to the server and stopping MySQL is usually the easiest way to allow a build through on a small stage server.

• The checkout goes in to /var/www/repo on the first stage server created.

• If the checkout failed because the git credentials are wrong, the git credentials go in a git store located at ~/.git-credential of the root user.

Triggering builds

Projects with cloud build config have a Build button. This build button is simply a link to a URL which can be used from any pipeline to trigger a build automatically, as the link does not require authentication as it contains a non-guessable randomly generated unique project key. As a build is performed on a stage server, updating the stage service itself is near instantaneous and happens as part of a build cycle.

If you click the button manually, don't close the page that opens as it will give you the build log in realtime.

Deploying builds

When you're ready to deploy, projects have a deploy button. Like the build button, this contains a link that can be used in external pipelines as well. This works by copying the build artifacts from the projects indicated build server (the first created stage server) and then uploading them to resources in the production environment via SSH. It waits for the upload to complete for every node before proceeding further. Once every node has a copy of the files, it then proceeds to one by one reload the services.

Deployment rotation

When Socialstack cloud deploys a project, it will move existing code out of the way by putting it into a directory called deploy which is usually found at either /var/www/stage/deploy or /var/www/prod/deploy. Note that it specifically does not do this with the Content directory as the Content directory is considered to be stateful. The previous build is put into a directory called prev. Like this:

• If deploy/prev-2 exists, delete it.
• If deploy/prev-1 exists, rename it to prev-2.
• If deploy/prev exists, rename it to prev-1.
• Create deploy/prev and move the previous build into it.

This means only the previous 3 deployments are retained and you can find them in the deploy folder should you need to recover them. Note that because this happens primarily using the move command rather than by copying it is usually an extremely fast switching operation.

Creating a server

Once you've created a project, you'll be able to start creating resources. They can be created in any order but it is suggested to create a singular medium size stage server first:

The available platform and region options are based on whichever platform(s) you have configured in your project.

Adding existing servers

This mechanism exists to add servers created manually/ externally. It is a way to use SocialStack Cloud features without providing an API key. It also works as a way to add things like managed MySQL instances via providing the username and password from the host.

How servers are deployed

This is a high level overview of the steps that SocialStack cloud goes through whilst deploying a resource. Because there is currently a very broad spectrum of combinations, it is suggested to either check the source or just deploy the combination you're interested in and then check its deploy log as it will list everything that it ran aside from the contents of uploaded files. There is a button to view the creation log on a server's information panel.

SocialStack Cloud will go through a series of phases when setting up a server. The current phase a server is in can be seen on a particular servers information screen via the project's page. The setup process will vary depending on which resource is being setup, which cloud and which operating system it is on.

Server creation

When the OS is unspecified, SocialStack cloud will deploy AL2 (Amazon Linux 2) on AWS and Ubuntu everywhere else. It will then wait for the server to exist and indicate that it is ready. It does this by connecting over SSH periodically as a user called "sscloud" to check the cloudinit status.

Installing software

When the resource is ready for installation, SocialStack cloud then executes install commands over SSH. On most SocialStack servers, this installs dotnet, Nginx and MySQL, then tunes the OS for typical SocialStack high volume throughput. The complete log of everything that is executed during server setup is available via the "view creation log" button on a particular resource.

DNS

By default all SocialStack cloud projects are given an automatically generated DNS subdomain under .stage.socialstack.dev and the DNS records are added to the DNS zone for socialstack.dev which is currently hosted on rackspace.

Known issue with the rackspace DNS zone: it is too large! Lately this has been surfacing as multiple failed certificate requests. You just have to be a little patient with it and retry.

Certificates

SocialStack cloud uses the DNS authorization technique to obtain a Let's Encrypt certificate for the site too. DNS authorization is useful for multiple reasons: firstly it is a wildcard certificate so fewer are needed; larger deployments can typically all use the same wildcard certificate for example. Secondly http based authorization can be challenging when loadbalancers are involved as the response will be directed to a random server.

It is able to perform the DNS authorization technique for any domain that it has record access to. Currently that is any domain on the rackspace account.

Once a certificate has been obtained, SocialStack cloud then checks if it can apply it anywhere. It will apply it if a particular resource has a matching environment and address. The certificate is applied by connecting to the server over SSH and then uploading the certificate in to /etc/ssl/cert and then reloading nginx (the web server). This certificate location can also be seen in the nginx configuration which is usually located at /etc/nginx/sites-enabled/ (this can vary between OS's but is very commonly here).

Certificates (Manual Steps with http access)

Here are the steps to manually create and install custom ssl certs, this may be required for multi-domain sites

Onetime install/update certbot

sudo apt install certbot python3-certbot-nginx

Create the certificates, adding your domain names as necessary

certbot certonly --nginx -d amb-sst.stage.socialstack.dev -d amb-amba-sst.socialstack.dev -d amb-bga-sst.socialstack.dev

The output from certbot should be something like -

	- Congratulations! Your certificate and chain have been saved at:
	   /etc/letsencrypt/live/amb-sst.stage.socialstack.dev/fullchain.pem
	   Your key file has been saved at:
	   /etc/letsencrypt/live/amb-sst.stage.socialstack.dev/privkey.pem
	   Your cert will expire on 2024-07-07. To obtain a new or tweaked
	   version of this certificate in the future, simply run certbot
	   again. To non-interactively renew *all* of your certificates, run
	   "certbot renew"

	 - Your account credentials have been saved in your Certbot
	   configuration directory at /etc/letsencrypt. You should make a
	   secure backup of this folder now. This configuration directory will
	   also contain certificates and private keys obtained by Certbot so
	   making regular backups of this folder is ideal.

So the two important files are

/etc/letsencrypt/live/amb-sst.stage.socialstack.dev/fullchain.pem
/etc/letsencrypt/live/amb-sst.stage.socialstack.dev/privkey.pem

Now just need to link them back into the nginx config

Check the site config to get the ssl config

more /var/www/prod/nginx.conf

so usually something like

ssl_certificate /etc/ssl/cert/fullchain.pem;
ssl_certificate_key /etc/ssl/cert/privkey.pem;

Next you need to use our new files to replace the existing files

cd /etc/ssl/cert

tar -cvf certs.tar *.pem

rm *.pem

ln -s /etc/letsencrypt/live/amb-sst.stage.socialstack.dev/fullchain.pem /etc/ssl/cert/fullchain.pem
ln -s /etc/letsencrypt/live/amb-sst.stage.socialstack.dev/privkey.pem /etc/ssl/cert/privkey.pem

So you should now have 2 symbolic link files to the new certs as below

-rw-r--r-- 1 root root 20480 Apr  8 08:33 certs.tar
lrwxrwxrwx 1 root root    65 Apr  8 08:34 fullchain.pem -> /etc/letsencrypt/live/amb-sst.stage.socialstack.dev/fullchain.pem
lrwxrwxrwx 1 root root    63 Apr  8 08:34 privkey.pem -> /etc/letsencrypt/live/amb-sst.stage.socialstack.dev/privkey.pem

Finally restart nginx

nginx -s reload

Certificates (Manual Steps for Cloudfare Hosted)

Here are the steps to manually create and install custom ssl certs, this may be required for multi-domain sites

Onetime install/update certbot

sudo apt install certbot python3-certbot-dns-cloudflare

Obtain an api token with the permissions from cloudflare

Zone: DNS:Edit
Zone Resources: Include the specific zone(s) for which you want to manage DNS records.

Create a config file containing the following

dns_cloudflare_api_token = YOUR_CLOUDFLARE_API_TOKEN

nano /etc/letsencrypt/cloudflare.ini
chown root:root /etc/letsencrypt/cloudflare.ini
chmod 600 /etc/letsencrypt/cloudflare.ini

Request the certificate adding your domains as necessary

certbot certonly \
  --dns-cloudflare \
  --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \
  -d www.test.com \
  -d test.com

The output from certbot should be something like -

	- Congratulations! Your certificate and chain have been saved at:
	   /etc/letsencrypt/live/www.test.com/fullchain.pem
	   Your key file has been saved at:
	   /etc/letsencrypt/live/www.test.com/privkey.pem
	   Your cert will expire on 2024-07-07. To obtain a new or tweaked
	   version of this certificate in the future, simply run certbot
	   again. To non-interactively renew *all* of your certificates, run
	   "certbot renew"

	 - Your account credentials have been saved in your Certbot
	   configuration directory at /etc/letsencrypt. You should make a
	   secure backup of this folder now. This configuration directory will
	   also contain certificates and private keys obtained by Certbot so
	   making regular backups of this folder is ideal.

So the two important files are

/etc/letsencrypt/live/www.test.com/fullchain.pem
/etc/letsencrypt/live/www.test.com/privkey.pem

Now just need to link them back into the nginx config

Check the site config to get the ssl config

more /var/www/prod/nginx.conf

so usually something like

ssl_certificate /etc/ssl/cert/fullchain.pem;
ssl_certificate_key /etc/ssl/cert/privkey.pem;

Next you need to use our new files to replace the existing files

cd /etc/ssl/cert

tar -cvf certs.tar *.pem

rm *.pem

ln -s /etc/letsencrypt/live/www.test.com/fullchain.pem /etc/ssl/cert/fullchain.pem
ln -s /etc/letsencrypt/live/www.test.com/privkey.pem /etc/ssl/cert/privkey.pem

So you should now have 2 symbolic link files to the new certs as below

-rw-r--r-- 1 root root 20480 Apr  8 08:33 certs.tar
lrwxrwxrwx 1 root root    65 Apr  8 08:34 fullchain.pem -> /etc/letsencrypt/live/www.test.com/fullchain.pem
lrwxrwxrwx 1 root root    63 Apr  8 08:34 privkey.pem -> /etc/letsencrypt/live/www.test.com/privkey.pem

Finally restart nginx

nginx -s reload

Managing Services

As the deployment can vary due to both the software being deployed and the OS it is being deployed on, this is just a rough guideline for common scenarios. As before it's recommended to take a look at the creation log to get a better idea of what software is actually running, particularly if the

SocialStack API on Ubuntu

SocialStack cloud installs a service called either "stage" or "prod" which is managed by systemD on ubuntu. This means you can restart it etc like so:

# superuser only
sudo su

service stage stop
service stage start
service stage restart

And view its current log using journalctl like so:

journalctl -u stage -n 100

Accessing a server

Because servers are deployed in to a "bring your own cloud" environment, there are two ways to access a server. Either through the underlying cloud platform if you have access to that, or via the "Let me in" button on a particular server's information panel. This button requires that your SocialStack cloud account has an SSH public key associated with it. This button tells SocialStack cloud to add your public key to the target server's user list if it wasn't already there, and SocialStack Cloud adds it via connecting over SSH to the target server.

When you use the "Let me in" button it will also provide a useful summary of the connection details.

Server health reporting

All resources created on SocialStack Cloud are configured with a ReportTo URL. This tells the resource to frequently ping SocialStack Cloud with a health update. The current list of healthy resources is used to configure loadbalancers.

SocialStack cloud retains historical health data in the form of (millions of) revisions of the "MonitoredHost" content type.

SocialStack Cloud Source

The SocialStack project is currently a private repository located here.