Refresh a Development Server in Azure

Approximately once every few months we rebuild our Development server with a fresh database clone of our Production environment. In addition to giving us current data to work with, this also lets us test plugins or other building elements on an identical setup to our Production install. Now for context, this recipe assumes you already have a Production and Development server running. It could be used to aid in the creation of a Development instance as well, but it does not cover the extra steps needed to have IIS host multiple sites on the same server.

Our environment is Azure based. We have a single Azure VM running both our Production and Development instances. Some would say this isn't best practice as one tanking could affect the other, but in practice we haven't found this to be the case. IIS splits the "IIS Worker" processes evenly and if one locks up, the other site is still functional. We also have separate AppPools setup for each which further isolates performance cross-contamination. So for us, avoiding the cost of a second server (which we used to have) is worth the risk of any potential complications that arise from sharing the hosting server.

In addition to our Azure VM, we also have Azure SQL Instances for our Databases. Our Production currently is a dynamically scaled S2-S6 instance using the "Azure Management" Plugin, and our Dev server lives on an S2 level.

Cloning the Database

First we need to create a new copy of your production Database. We use Azure SQL Instances, so I will focus on that here. In Azure they use the "Restore" function to create a copy of a DB. It doesn't overwrite your production DB (despite the name) rather it creates a new DB with the name you assign it, and fills it with the data from the one you are "restoring."

(Note from Future Brent: Post writing this recipe, I've started to use "Copy" instead. Copy is MUCH FASTER than "Restore" as it copies from the live DB at the current time vs restoring 5min old data from some backup server that must be slower... While a "Restore" can take 45min, but a "Copy" takes about 6 for us. The rest of the process is the same for steps 1-3 except you aren't asked to specify a time to restore from as it takes the current time for a "Copy".)

1) Log into Azure at https://portal.azure.com and navigate to your SQL Databases. Choose your Production DB you wish to clone and then select "Restore" at the top right menu (or "Copy" to the left of it here).

2) In the prompt that opens, pay attention to three things.

• First, make sure your are restoring to the latest (it does by default) possible data from the production DB. Time is in UTC here. This is what Azure calls a "Point-In-Time" restore, so you can pick a time down to the millisecond of when you want the data from... but it is only applicable from the last time a log file was backed up, which happens every 5 minutes. So your "latest" may be 5 minutes old, or say 1 minute old, if that is where your are loading this from since the last log file backup.
• Second, you need to enter the name of your new DB. You can leave it as the default, but I think a more descriptive name is helpful. You'll need this later.
• Third, you can set the "level" of the DB here. By default it will use whatever settings you have on the source DB you are Restoring. If you normally run your Production at say "S4" level, but your Dev at "S2" level, you COULD change it to "S2" here. I would advise against that however, as having the faster speed will help for our rebuild, and then once we are complete, we will come back and scale it down to "S2" at the end to save on costs.

3) Once you are happy, click "Review + create" and then "Create" after confirming your settings. You will now see "Deployment is in progress" and this can take awhile depending on your DB size. Our DB is around 50GB and it isn't uncommon for deployment to take >1hr.

Cloning the Files

While this is deploying, we can set up the file system. Log into your Production copy of Rock and clear the cache. This will create less data to copy on your Web Server.

4) Click the "i" at the bottom right of your Rock internal website.

5) Click the "Clear Cache" button and after you get the success message, click "Done".

Next we will remove our old file system for our Dev server and replace with with a fresh file set from Production. This will ensure we have an exact match, including any version specific changes or uploads that have happened on production. After the copy, we will change the database the Dev server looks at.

6) On the Web Server, Open (Internet Information Server) IIS and stop the Dev site and its App Pool.

7) Delete the contents of your Dev server file system (in our case, this is in C:\inetpub\wwwroot-dev) and then copy and paste the contents of your production site's file system into the Dev folder.

Point the Install to the new DB and do Pre-Flight checks

8) Open the new copy of "web.ConnectionStrings.config" that is in your dev folder (in our case, C:\inetpub\wwwroot-dev\web.ConnectionStrings.config) and change the text after "Catalog=" to be your new dev database name from step 2. If you can't save the file here, you may not have stopped the AppPool in step 6. Or if you have some other permissions issue, try saving it to your desktop instead, then copy/paste it over the one in the dev folder and say yes to the security prompt.

9) Now would be a good time to look on your production site to see which Jobs you want to still run on your Dev site. We will be disabling all jobs except for a list we want active, so let's check that out to make sure your list matches mine. If your job IDs are different, make note of that. Click "Admin Tools - System Settings" and choose "Jobs Administration" to see your list. Go through and decide which ones you want to still run on Dev. If you want to keep it, click into the job and make note of the ID in the URL path.

Don't assume your IDs are the same as ours, but for comparison's sake, our list of jobs we keep running on Dev are:
1 Job Pulse
7 Rock Cleanup
8 Process Workflows
14 Calculate Metrics
23 Spark Link
25 Database Maintenance
29 Universal Search Re-Index
45 Calculate Person Signals
46 Update Persisted DataViews
74 Campaign Manager
228 Steps Automation

10) Now would be a great time to check if the Azure DB Restore is complete. If you don't see a new DB in your SQL Databases, it is not. Go have a coffee and come back later. If it is, we can proceed. YOU WILL WANT TO WAIT to do the next step until the SQL DB is there. If you don't wait, the site will "go live" whenever it finishes restoring and if you aren't paying attention and able to quickly get on and change things, your Dev site may start doing things you don't want it to do (like sending communications).

Bring it to life

10) Once the restore is complete and the SQL table appears on Azure (which may take an hour or so), use IIS to start Dev AppPool and site.

Clip its wings

11) Open a browser and navigate to your Dev site. Once the site reboots (give it a couple minutes to spin and it will load eventually), log in and disable all communication transports! Go to "Admin Tools - Communications - Communication Transports" and for any that are marked as "Active" (in our case, Mailgun and Twilio) open them and change the "Active" drop down to "no" and save them.

12) We need to disable all jobs that send out data or sync with external services like PCO or Pushpay or Google Location Services in Admin Tools - System Settings - Jobs Administration. Pretty much anything that says it syncs out or sends things. But there are a few we want to still leave active but change their notifications to "none" so we are not hit with errors on a slower DB or whatnot. This could take A LONG TIME to go through each job manually, so let's automate this with some SQL. As noted in step 9, we are going to disable ALL jobs except for the following job IDs/Names which will be left active with notifications off:
1 Job Pulse
7 Rock Cleanup
8 Process Workflows
14 Calculate Metrics
23 Spark Link
25 Database Maintenance
29 Universal Search Re-Index
45 Calculate Person Signals
46 Update Persisted DataViews
74 Campaign Manager
228 Steps Automation

Open "Admin Tools - Power Tools" and choose "SQL Command". Be sure to use your list of Job IDs if it is different than ours in this script. Copy/Paste this into the SQL box and press Go.

You can check if this worked as intended by going to "Admin Tools - System Settings" choosing "Jobs Administration" and sorting by "Active" descending. It should match your intent. If not, feel free to manually set jobs to active as needed.

Brand the site as a Development Environment

13) Now copy/paste this SQL into the SQL Command block from the last step and run it. This will add the DEV warning to the header of all pages. After doing this, you will need to clear the site cache again (see steps 4 and 5) and then refresh the page to see the result.

14) Go to "Admin Tools - General Settings" and open "Global Attributes" and update the "Internal Application Root" and "External Application Root" to be the dev addresses.

15) Add a new HTML block to your Internal Home page to inform users when this development data was created. Click on "Page Zones" in the bottom right menu, edit the "Main" zone, add an HTML block and give it a name (I'm just using "HTML" as the name) and move it to the top of the stack if you have multiple items there.

Click the "Block Configuration" item in the bottom right menu, edit the newly created "HTML" block and copy/paste the following into it, updating the date to match your Dev build date.

16) Go to "Admin Tools - CMS Configuration" and choose "Themes" so we can change the Brand Color of your active theme. In our case we are using the "Rock" theme. So we click "Rock" and change the theme "Brand Color" to be a color VERY DIFFERENT from our normal branding. In our case, we use #8b0000. We also use CSS Overrides for the Rock theme to further customize our install, so we need to change some of our colors there as well:

(We will also repeat this "Brand Color" change on the theme we use for our External Rock site... not documented here.)

Add visual cues to External site

17) Go to your external site on your dev server and log in. Edit the top left block Edit https://devmy.cedarcreek.tv blocks and change header text HTML to:

18) For fun, we also add an HTML block to our Dev site's login page that gives further visual confirmation you are on the Dev site. So we go to our Dev URL /page/3 and add an HTML block above the login block, and insert this code:

Click the "Page Zones" icon and edit the "Main" zone.

Click the plus sign to add a new section, select "HTML Content" and name it something like "HTML" and then sort it to be the top item.

Click the "Block Configuration" icon and edit the new HTML block you added. Insert the code here.

Customize Check-In for Dev

19) On our setup we have a custom theme we use for check-in on our Development server. This allows us to train volunteers on non-live data. But to make it obvious our check-in stations are on the dev site, we need to change what theme they use. So we go to "Admin Tools - CMS Configuration" and choose "Sites" and change the "Rock Check-in" and "Rock Attended Check-in" (from a plugin) sites to use our Demo theme.

20) We also have a 24/7 schedule available for check-in training on our Dev server. So we need to set that as the active schedule for our Kids/Students/Volunteer areas. So we go to "Admin Tools - Check-in" and choose "Check-in Configuration" (repeat for each area: Kids, Student Ministries, and Volunteer). Click schedule button, Drop down "Schedule Category" to "testing schedule" and click select all buttons for the test time column and hit save at the bottom. (If a "no schedules" error appears, go into "Checkin - Schedules - Testing Schedule" and make sure the 6:00 is active.) Repeat for all areas you need to update...

Enable MailTrap

21) We use the MailTrap SMTP service to test emails on our Dev server. If you don't know about MailTrap, check out the recipe explaining it here: MailTrap Email Testing

We need to first enter our credentials into the SMTP Communication Transport if not there already (we keep ours in production with that transport disabled, so any time we clone to Dev, they are already there and only need enabled). Click into "Admin Tools - Communications" and choose "Communication Transports" and edit the SMTP transport, set active to "yes" and enter your mailtrap info if needed and click save.

22) Now that the Communication Transport is active, we need to assign it to the Email Communication Medium. Go to "Admin Tools - Communications" and choose "Communication Mediums" and pick "Email" from the options there. Drop down the blank "Transport Container" and set it to "SMTP" and click save. Change Admin Tools - Communications - Communication Mediums - Email to use SMTP for the Transport Container.

Rock Shop Notifications

23) We use the "Rock Shop Notifications" plugin on our Dev server to see what new plugins come out and to stay informed of updates. We don't run this on our Production server, so we need to go tot he Rock Shop and install the "Rock Shop Notifications" Plugin. After installing, you have to go to "Admin Tools - System Settings" and choose "Job Administration" and edit the job "Check for new plugins" and click save without changing anything. Manually run the job once to get it started. The results will now be on your Internal Rock home page.

24) In the Rock Shop, install any other plugins that need installed on your Dev environment. We have a few we are testing on Dev that we don't have on Production and as such, need to re-install them when we rebuild Dev, similar to the above case of the "Rock Shop Notifications" plugin that only lives on our Dev environment.

Clean up Azure

25) Now that you are done rebuilding Dev, let's clean up Azure by deleting the OLD Dev DB (so you aren't paying for a DB you aren't using). On your SQL Databases tab in the Azure Portal, check the box next to the OLD Dev DB, and click the "Delete" button at the top right. Type "yes" where prompted to confirm and click "Delete" at the bottom. This may take a couple minutes to process.

26) Lastly, we will scale down the new Dev SQL Instance to S2 speed (or whatever you use) to save money on our monthly invoices. Choose the Dev SQL DB from your list and click on "Compute + Storage" on the left, scale down your Instance as needed (we are dropping from 200dtu which is S4, to 50 dtu which is S2 level) and press "Apply" at the bottom. Give it a minute to process and your changes should be reflected in the SQL Databases page.

Well there you have it! A detailed step-by-step of how to rebuild a Dev server with new Production data on Azure. Keep in mind, this seems like a lot, but I can usually knock it out in about 30 min or less since I am familiar with the steps. I hope this helps!