Years ago I stumbled on a great blog about SSIS best practices and naming conventions. In the years to come I could refer back to this at the start of every SSIS project. Written by Jamie Thomson, this has become the standard, and although there were variants, Jamie’s still remains very popular (Jamie Thompson, Link).

I want to propose a list of best practices and naming conventions for the various elements of ADF based around my experience with ADF. This list may change over time as I discover what works and what does not. At present the standard names generated by ADF are generic and not very helpful (LinkedServiceAzureSQLDataWareouse1 etc). In light of this I propose the following best practices and standards for naming ADF elements.

 

Best practices

 

1. Use a standard naming convention. For a large ADF project you can end up with 300-400 json objects. It is tricky to navigate that many objects and deploying from visual studio makes this even harder. Having a standard helps to quickly locate which json object needs to be changed.
2. When you have a solution with a lot of JSON objects (a few hundred or more) publishing from visual studio becomes an issue. A bottleneck begins to appear, which is brought on by each element needing to validate against it peers. When you have hundreds of elements deploying new/updated pipelines adds a significant overhead. In my recent example I had 222 JSON objects and 74 pipelines, when initially deploying, each pipeline (after solution validation) took ~10 seconds each. Once published and updated this increased to ~40 seconds per pipeline.

 

74×10 is approximately 12:20 minutes (13 minutes in total)

 

74×40 is approximately 49:20 minutes (gave up after 40 minutes)

 

This can be resolved by adding a step in to your automation which removes your ADF instance before publishing a large amount of changes (see point 3 – Automate with Po$h)

 

3. Don’t neglect PowerShell. PowerShell is a great way to automate the development of json objects – especially when you need to create a few hundred! I have a blog coming in the next few weeks detailing how to do this.

 

* More best practices will be added.

 

Naming conventions

 

There are a few standard naming conventions which apply to all elements in Azure Data factory.

 

1. Object names must start with a letter or a number, and can contain only letters, numbers, and the dash (-) character.
2. Maximum number of characters in a table name: 260.
3. Object names must start with a letter number, or an underscore (_).
4. Following characters are not allowed: “.”, “+”, “?”, “/”, “<”, ”>”,”*”,”%”,”&”,”:”,””

 

(Microsoft, 2017 [https://docs.microsoft.com/en-us/azure/data-factory/data-factory-naming-rules])

 

Linked services & Datasets.

 

A linked service connects data from a source to a destination (sink), it stands to reason that there would therefore be the same for a data set. Rather than having two separate lists, you will see the below table has a column for Linked services and datasets.

 

Pipelines

 

Pipelines are slightly different in that there are two different types. The copy data activity and the data transformation activity.