Jason's Blog

An Example using Zillow Data In order to identify outliers (outlying Y), we can take look at the values of semistudentized residuals, studentized residuals, studentized deleted residuals. a. semistudentized residuals: $\frac{e_i}{\sqrt{MSE}}$. b. studentized residuals: $\frac{e_i}{\sqrt{MSE(1-h_{ii})}}$, where $h_{ii}$ is diagonal element of Hat matrix. c. studentized deleted residuals: $d_i = Y_i - \hat{Y_{i(i)}} = \frac{e_i}{1-h_{ii}}$, redo the regression without observation i, to get $MSE_{(i)}$, so that $s^2\{d_i\} = MSE_{(i)}(1+X'_i(X'_{(i)}X_{(i)})^{-1}X_i)=\frac{MSE_{(i)}}{1-h_{ii}}$ $t_i = \frac{d_i}{s\{d_i\}}=\frac{e_i}{\sqrt{MSE_{(i)}(1-h_{ii})}}\sim t(n-p-1)$ Outlying X ( leverage ) can be identified by using Hat matrix. An observation is usually considered to be a leverage if $h_{ii} > 2p/n$. Another suggested guideline is that $h_{ii}$ exceeding 0.5 indicates very high leverage, whereas between 0.2 and 0.5 indicates moderate leverage. Influential c...

Absenteeism . Data on 77 employees of the ABX Company have been collected. The dependent variable is absenteeism (ABSENT). The possible explanatory variables are COMPLX = measure of job complexity SENIOR = seniority SATIS = response to "How satisfied are you with your foreman?" In this example, use SENINV = 1/SENIOR, which is the reciprocal of the seniority variable, and COMPLX as two of the explanatory variables. The variable SATIS should be transformed into indicator variables (1 is very dissatisfied, 2 is somewhat dissatisfied, 3 is neither satisfied nor dissatisfied, 4 is somewhat satisfied, 5 is very satisfied). Q1 . Is there a difference in average absenteeism for employees in different supervisor satisfaction groups? Q2 . Using the model chosen, what would be your estimate of the average absenteeism rate for all employees with COMPLX = 60 and SENIOR = 30 who were very dissatisfied with their supervisor? What if they were very satisfied with their supervisor...

175. Combine Two Tables Table:  Person +-------------+---------+ | Column Name | Type | +-------------+---------+ | PersonId | int | | FirstName | varchar | | LastName | varchar | +-------------+---------+ PersonId is the primary key column for this table. Table:  Address +-------------+---------+ | Column Name | Type | +-------------+---------+ | AddressId | int | | PersonId | int | | City | varchar | | State | varchar | +-------------+---------+ AddressId is the primary key column for this table. Write a SQL query for a report that provides the following information for each person in the Person table, regardless if there is an address for each of those people: FirstName, LastName, City, State My Solution: # Write your MySQL query statement below select FirstName, LastName, City, State from Person left join Address on Person.PersonId = Address.PersonId; 176. Second Highest Salary Write a SQL query to get the ...

Unfortunately, there is no weighted built-in functions in Python. If we want to get some weighted percentiles by Python, one possible method is to extend the list of data, letting the values of weight as the numbers of elements, which is discussed in a Stack Overflow poster . For example, if we have a data like, score   weight 5          2 4          3 2          4 8          1 we firstly extend the list of scores to {5, 5, 4, 4, 4, 2, 2, 2, 2, 8}, and then find the percentiles such as 10% or 50% percentile. The limitations of this method are, (1) weight must be integers; (2) values of weight cannot be very large. What if we want to calculate the weighted percentiles of a large dataset with very large non-integer weights? In this article, I want to show you an alternative method, under Python pandas. step1: given percentile q, (0<=q<=1), calculate p = q * sum of wei...

In the last post "The Difference of Occupation Choice Among Graduates with Different Majors and Degrees", I built a logistic model with log of salary as one of the independent variable. But we will concern about the endogeneity of it, because the salary may related to other unincluded variables that may affect the probability of working at education institute, and moreover, as a current salary, the occupation itself may affect the salary. So some instruments are needed to solve the possible endogeneity problem in the regression analysis. In the dataset, a variable "satis" measuring the satisfaction of salary seems to be a good possible instrument. For this variable, 4 means very satisfied, 3 means somewhat satisfied, 2 means somewhat unsatisfied, and 1 means very unsatisfied. Let's firstly take a look at the model without using IV. The dependent variable is whether working at education institutes, and independent variables are degree, major, number of years af...

The percentages of graduates who get work in educational institutes, industry, or government are different among them with different majors and degrees. This article tries to take a look at how the majors and degrees, and other factors may affect people's occupation choices. The data is from 2013 National Survey of Graduates  with totally 104599 observations and 515 variables. A subdata is extracted with 87145 observations who had graduated before 2013 and currently had jobs during the survey reference month (Feb 2013). There are 9 interested variables: Variable in the Raw Data Description Variable Used in Regression Models Catogories emsecsm Employer sector: 1 = education institute, 2 = government, 3 = industry job_edu 1 = education institute, 0 = others dgrdg  Degree: 1 = bachelor, 2 = master, 3 = PhD, 4 = professional degree 1 = bachelor, 2 = master, 3 = PhD or professio...