What order does PROC SQL do your work?

To optimize a Structured Query Language (SQL), the database professional must be friend this order and perhaps even embrace it.

The best onlie and class room training center for SAS, Signetsoft, Bangalore. 

Who is your best friend? I’m talking about the order in which SQL processes your statements. Simply put, in what order does SQL do your work? (From my previous post you know another SQL rule, the order in which YOU submit statements to SQL.) I just want us to be clear that these are two very different orders.

First up, what makes SQL different from other programming languages? It’s the way SQL processes code. Most programming languages including SAS start at the top of your code and make their way to the bottom. However, SQL processes them in a unique order, known as Logical Query Processing Phase. This phase (also referred to as a clause) generates a series of virtual tables with each virtual table feeding into the next clause or phase. And no, we can’t view these virtual tables, sorry about that.  That’s just the way SQL works!

When I shared the processing order with my recent SQL1 class, Lindsay Jacks of Cancer Care Ontario asked if I would blog about it.  I hope this sharing will help breakdown any surprises you may expect to see when you submit SAS code. And in turn help in understanding how to optimize SQL queries.

Here then is the logical query processing phase order.

1.  FROM

The 1st clause SQL processes is the FROM. It tells SQL where to grab your tables from. Given that this clause is the first to execute, it’s also our first chance to boil down your table sizes. From an efficiency perspective, this is why we may need to think about putting as many ON clauses as possible on JOINs versus putting them on the WHERE clause.

2.  WHERE

The 2nd clause SQL processes is the WHERE.  The WHERE clause pre-processes data. It selects just the rows that meet the WHERE criteria.  The results of WHERE clause processing are stored in an intermediate table.

SQL coders who have sometimes been surprised to see the WHERE clause fail may now have their answer. In the example that follows, the WHERE has no idea how to filter Bonus and doesn’t know what Bonus means. Just because we constructed Bonus on the SELECT doesn’t mean anything to the WHERE as the SELECT is one of the last clauses in the processing order & yet to be looked at by SQL. Consider the following example where the Log rightly complains:

43     /* Try to subset by referencing a new column with its alias */
44   proc sql;
45      select Employee_ID, Employee_Gender, Salary,
46             Salary * .10 as Bonus
47         from orion.Employee_payroll
48         where Bonus < 3000;
ERROR: The following columns were not found in the contributing tables: Bonus.
49   quit;
NOTE: The SAS System stopped processing this step because of errors.
NOTE: PROCEDURE SQL used (Total process time):
      real time           0.01 seconds
      cpu time            0.01 seconds

3. GROUP BY

The 3rd clause SQL processes is the GROUP BY.  The role of the GROUP BY is to get your data into groups. To do this grouping, sometimes SQL may need to order your data, but there’s no guarantee that data will be in ascending or descending order. Explicitly placing an ORDER BY clause is our only guarantee that data will be in order.

4. HAVING

The 4th clause SQL processes is the HAVING.  This clause subsets the groups created by the GROUP BY clause based on the HAVING clause predicates, and, like the WHERE clause, builds an intermediate table.

5. SELECT

The 5th clause SQL processes is the SELECT. This is what tells SQL what columns to pull into the query, existing or calculated. Note: Columns are calculated on the SELECT only one time regardless of how often they are referenced, whether on the WHERE (with the calculated keyword) or the ORDER BY (without the calculated keyword). The SELECT statement is able to use the results of anything it calculated in other instances using the same calculation.

6. ORDER BY

The 6th clause SQL processes is the ORDER BY. By now it should come as no surprise that the ORDER BY has access to columns created on the SELECT but the WHERE doesn’t (see 2 above to as a reminder).

Any earlier confusion about the calculated SELECT columns (see 2 again) not being available in the WHERE clause, but available in the ORDER BY clause would now make perfect sense. Because the SELECT is executed before the ORDER BY, all columns in the SELECT will be available at the time of ORDER BY execution. So the following code works perfectly:

68   proc sql;
69      select Employee_ID, Employee_Gender, salary,
70             Salary * .10 as Bonus
71         from orion.Employee_payroll
72         order by 4;
73   quit;
NOTE: PROCEDURE SQL used (Total process time):
      real time           0.03 seconds
      cpu time            0.03 seconds

Itzik Ben-Gan makes a good point with a poster (i can provide in my next post) that captures the processing order.

Why do I call this processing order your best friend in the database world? Because we have likely at some time come across never ending queries and not known how to optimize. Here then is a starting point for us to try to keep in mind next time we submit a query, so we can modify the way we optimize SQL queries.

(courtesy : sasblogs)

www.signetsoft.com

The best class room and online training center for SAS, Signetsoft, Bangalore. 

All the classes are fully practical and 100% job oriented.

We do provide training in SAS/BASE and Advanced.

Corporate / Class room / Online / weekend / weekdays / fast track trainings are available.

Attend for demo and 1st day class then you can come to one decision on payment of fee.

SAS courses available for Certification, Internship, as an academic project etc…

Signetsoft is specialized in SAS CLASS ROOM and ONLINE training classes.

And SIGNETSOFT is located in Marathahalli, Bangalore. If it is not possible due to distance and traffic to reach, then we do provide ONLINE classes too.

So far we had given classes for abroad students (Majorly USA, UK, Canada, Australia, Singapore, Dubai, etc).

Especially for students from Malleswaram, Jayanagar, JP Nagar we do provide special ONLINE batches.

And we are happy to announce that very soon we are going to open new branches in JP Nagar and Malleswaram.

SAS training center (Signetsoft)

Contact us:

www.signetsoft.com

info@signetsoft.com

mob: 98 4455 9330

Ph: 080-41 666 550

SIGNETSOFT is the best Training center in Bangalore for SAS, SAS clinical, SAS projects, SAS Certification, Advanced Excel, VBA, Java, Clinical Data Management.

A practical approach! and real time expert faculty, good in placement record.

contact us: info@signetsoft.com

mob: 9844559330

Phone: 080-6547 2060

Website: http://www.signetsoft.com

Thanks for visiting our blog.

The better Way to become THE BEST in SAS

http://www.signetsoft.com

 SAS is an analytical tool, programming language, data manipulation tool, reporting tool. So to become the BEST is SAS, need the following:

• As a language, it needs practice for logical thinking
• As an analytical tool, it needs much practice with multiple case studies
• As a data manipulation tool, it needs much practice on case study with real data
• As a reporting tool, it needs much practice with multiple reporting scenarios

Are all the above needs are possible by the following?

• Explanation on a white board by a SAS trainer (who doesn’t have practical expose in the industry)
• 2 to 24 Hrs of lab facility (with assistance of a senior student who recently finished his SAS course there itself)
• One photo copy of the some SAS prescribed book (like Little SAS, SAS by example etc.)
• One dummy / sample CV (to prepare your CV).
• Sample interview questions (the same which are available in internet from past 10 years)

—NO—

It is not at all enough!!!

Now a days in the interview, you won’t get questions like:

1. What are INFILE / INPUT?
2. What is the difference between Missover and Truncover?
3. How to sort the data in SAS?
4. How to remove the duplicates in SAS?
5. What is merge?
6. What Proc Append will do?
7. What ODS stands for?
8. How to extract data from EXCEL to SAS?
9. What are the types of macro variables?
10. Difference between NODUP and NODUPKEY

The above sample questions are almost 4 to 5 years back were there.

Now the companies are expecting the practical scenarios.

No definitions!

No formulas!!

No syntaxes!!!

The required things are:

Case studies, Scenarios, Comparisons, Macro / Automation related…etc.

Now, some sample questions for the base level (with SAS certification)

1. How to take all the duplicate records into a SAS data set?
2. NODUPREC  vs  DISTINCT *
3. X=month(today());         put x date9. ;     What is x value printed in log?
4. How can I save all my log /output into some external file automatically?
5. %LET vs Call Symput
6. What is the minimum length of a macro variable?
7. Where macro variable does gets stored?
8. How can I use a variable which is created in a data step into another data step?
9. How do you deal with huge data?
10. How can you speed up your code execution?

If you are trying to learn SAS to get a job, then learn SAS as a PRACTICAL subject. Then only  you can answer all the above kind of questions by yourself.

If you are experienced SAS programmer, then the questions will be different. The standards might be increased and sometimes they may ask same tricky questions (just to clarify whether you have real time experience or not in SAS)

Some sample questions:

1. How can I take duplicate records in to SAS data set except first and last of each unique value?
2. How can I take all the variable names of the data set into another SAS data set variable?
3. How can I get the list of files existing in the physical directory?
4. How can I export multiple SAS datasets into an EXCEL workbook?
5. Call Symput Vs Call Symputx? And what is the minimum length of a macro variable?
6. Can you write a macro inside another macro?
7. How many macros you will create a day?
8. What is the maximum size of data that you handled? And how much time it will take to process the data?
9. To whom do you report?
10. How do you get your daily work?

 

And after your SAS course if you are planning to learn a SAS Project you must be able to answer all the above questions.

If not, think about the another Trainer/ Training institute

Do not compromise for the fee….! What is the use of learning even if it taught for free and you can’t get the Job with it?

SO….try for SAS JOB COURSE, NOT just for SAS COURSE!!!

• Just by seeing the Face Book likes on the website, don’t fall in their magic!!!
• If the training centers are not allowing you for demo & 1st class before paying the fee then better you think twice…!

• At least have the opinion with other old students.
• If they say “OK we can proceed” then that’s not enough for your goal.
• If they say “YES, it is perfect” then you can agree…!
• Otherwise just attend the demo or the 1st class to decide and then pay the fee…!

SO FINALLY TO BECOME THE BEST IN SAS, YOU NEED:

• Just copying from white board is not enough.

100% practical approach (Each and every program you need to execute)

• No use of 24 Hrs lab facility

you need to do it practically in class itself!

•  Photocopy of SAS book is not enough

In that all the topics must be covered in your syllabus and you must have your class examples/programs for all those topics.

•  Dummy /sample CV is not enough

You must specify / highlight your SAS and other skills in your CV.

And also remember you are responsible for each and every term/keyword in your CV

• Sample interview questions are not enough

You need to have mock interviews to know where you are

 If you can get all the above, then you can join in that SAS training center with 100% confidence.

(Some training centers are giving SAS course even for Rs.5,000/-  with 25% of subject. And some other famous centers are charging in lacks of rupees with 50% of subject. So irrespective of fee structure, check for the quality.

If the fee is too low(6k, 8k,..), obviously you can’t expect quality.

If it is too high (more than one lakh also, don’t believe blindly.)

Just take actual opinion of old students or attend the classes yourself.

And now we want to tell you one thing.

 We very much happy and proud to say that, we do provide the above quality of training in our SIGNETSOFT.

SIGNETSOFT is not just a training institute. It’s a division of DAMA SIGNET SOLUTIONS Private Limited company.

DAMA SIGNET SOLUTIONS is a consulting and software development company.

We do provide training in SAS/BASE and Advanced.

Corporate / Class room / Online / weekend / weekdays / fast track trainings are available.

Attend for demo and 1st day class then you can come to one decision on payment of fee.

SAS training in bangalore

SAS courses available for Certification, Internship, as an academic project etc…

Signetsoft is specialized in SAS CLASS ROOM and ONLINE training classes.

And SIGNETSOFT is located in Marathahalli, Bangalore. If it is not possible due to distance and traffic to reach, then we do provide ONLINE classes too.

So far we had given classes for abroad students (Majorly USA, UK, Canada, Australia, Singapore, Dubai, etc).

Especially for students from Malleswaram, Jayanagar, JP Nagar we do provide special ONLINE batches.

And we are happy to announce that very soon we are going to open new branches in JP Nagar and Malleswaram.

SAS training center (Signetsoft)

Contact us:

www.signetsoft.com

info@signetsoft.com

mob: 98 4455 9330

Ph: 080-41 666 550

How does IF in SAS treat a missing value?

Every programming language has an IF-THEN statement that branches according to whether a Boolean expression is true or false. In SAS, the IF-THEN (or IF-THEN/ELSE) statement evaluates an expression and braches according to whether the expression is nonzero (true) or zero (false). The basic syntax is

if numeric-expression then
 do-computation;
 else
 do-alternative-computation;

One of the interesting features of the SAS language is that it is designed to handle missing values. This brings up the question: What happens if SAS encounters a missing value in an IF-THEN expression? Does the IF-THEN expression treat the missing value as “true” and execute the THEN statement, or does it treat the missing value as “false” and execute the alternative ELSE statement (if it exists)?

data A;
input x @@;
if x then Expr="True "; 
     else Expr="False";
datalines;
1 0 .
;

proc print noobs; 
run;

 

Ah-ha! SAS interprets a missing value as “false.”

and also check the result for the below program:

data A;
input x @@;
if not x then Expr="True "; 
     else Expr="False";
datalines;
1 0 .
;

proc print noobs; 
run;

Ah-ha! SAS interprets a missing value as “true.”  

More correctly, here is an excerpt from the SAS documentation:

SAS evaluates the expression in an IF-THEN statement to produce a result that is either non-zero, zero, or missing.
A non-zero and nonmissing result causes the expression to be true; a result of zero or missing causes the expression to be false.

If you do not want missing values to be treated as “false,” then do not reference a variable directly, but instead use a Boolean expression in the IF-THEN statement.

For example, in the following statement a missing value results in the THEN statement being executed, whereas all other numerical values continue to behave as expected:

Have you encountered places in SAS where missing values are handled in a surprising way? Please post your favorite example in the comments.

(courtesy: sasblogs)

The best Training in Bangalore for SAS, SAS clinical, SAS projects, SAS Certification, Advanced Excel, VBA, and for .Net     is Signetsoft. 

A practical approach!

Real time expert faculty!!

Good in placement record!!!

contact us: info@signetsoft.com

mob: 9844559330

http://www.signetsoft.com

A quick tour of SAS operators

I want to understand the logic behind the answer to the following question:

The following DATA step is submitted:

data one;
 x=3;
 y=2;
 z=x**y;
 run;

What should be the value of the variable z in the output data set? Will you please explain what the ** symbol means, and how we solve the question?

-Too Many Asterisks

It is not intuitive that the ** symbol means exponentiation, as in X to the Yth power.  There are several arithmetic, comparison, and logical SAS operators (symbols), as well as miscellaneous operators for use in SAS functions and for grouping.  Let’s take a quick tour.

Arithmetic Operators

Symbol	Definition	Example	Result
**	exponentiation	a**3	raise A to the third power
*	multiplication1	2*y	multiply 2 by the value of Y
/	division	var/5	divide the value of VAR by 5
+	addition	num+3	add 3 to the value of NUM
-	subtraction	sale-discount	subtract the value of DISCOUNT from the value of SALE
1 The asterisk (*) is always necessary to indicate multiplication; 2Y and 2(Y) are not valid expressions.

Note:  If a missing value is an operand for an arithmetic operator, the result is a missing value.  You can use parentheses ( ) to control the order in which the expression is evaluated.

Comparison Operators

Symbol	Mnemonic Equivalent	Definition	Example
=	EQ	equal to	a=3
^=	NE	not equal to1	ane 3
¬=	NE	not equal to
~=	NE	not equal to
>	GT	greater than	num>5
<	LT	less than	num<8
>=	GE	greater than or equal to2	sales>=300
<=	LE	less than or equal to3	sales<=100
	IN	equal to one of a list	numin (3, 4, 5)
1 The symbol that you use for NE depends on your personal computer.
2 The symbol => is also accepted for compatibility with previous releases of SAS. It is not supported in WHERE clauses or in PROC SQL.
3 The symbol =< is also accepted for compatibility with previous releases of SAS. It is not supported in WHERE clauses or in PROC SQL.

Note:  You can add a colon (:) modifier to any of the operators to compare only a specified prefix of a character string (name =: ‘St’ would give you all the values of the name variable that start with a capital S followed by a lower case t).

In addition to the above comparison operators, there is the MIN (><) operator which returns the lesser of the two values. The MAX (<>) operator returns the greater of the two values. For example, if A is less than B, then A><B returns the value of A (the minimum or lesser value).

 Logical Operators

Symbol	Mnemonic Equivalent	Example
&	AND	(a>b & c>d)
|	OR1	(a>b or c>d)
!	OR
¦	OR
¬	NOT2	not(a>b)
∘	NOT
~	NOT
1 The symbol that you use for OR depend on your operating environment.
2 The symbol that you use for NOT depends on your operating environment.

Note:  Logical operators are also called Boolean operators.  For an AND expression to be true, both sides of the expression must be true (a must be greater than b as well as c must be greater than d, above).  For an OR expression to be true, only one side of the expression must be true (a must be greater than b or c must be greater than d, above).  The NOT logical operator is my favorite.  (Just listen to the name:  The not logicaloperator!  It’s what I tell my husband I’m being when I’m not following his rules of logic.)  Anyway, the NOT logical operator reverses the logic of the AND or OR expression (a cannot be greater than b, above).

Other Useful Symbols

The other useful symbol is the concatenation operator, which is represented by two vertical bars (||, Windows), two broken vertical bars (UNIX), or two exclamation points (!!).  Do remember, however, that the concatenation operator does NOT trim leading or trailing blanks from character values, which can lead to some bizarre results!

I hope this brief tour of some of the most used symbols in SAS code has been helpful.  I’m off now to use my NOT logical operator to good advantage. By the way, the answer to the question at the beginning is z is equal to 3 to the 2nd power or 9 (z=x**y;).

(courtesy: sasblogs)

SIGNETSOFT is the best Training center in Bangalore for SAS, SAS clinical, SAS projects, SAS Certification, Advanced Excel, VBA, Java, Clinical Data Management.

A practical approach! and real time expert faculty, good in placement record.

contact us: info@signetsoft.com

mob: 9844559330

Phone: 080-6547 2060

Website: http://www.signetsoft.com