Wednesday, April 18, 2018

Advanced Topics in Python

 Advanced Topics in Python

Topics for Advanced Python usage:
Design Pattern - Using decorators, constructors, classes and data structures in Python
Using Flask framework in the same way as React using the same folder config and other settings. In place of JS we will use Python
Functional Programming in Python and passing on functions in a function. More list comprehensions.


__init__

single underscore vs double underscore


Python Generators and Iterator Protocol
Python Meta-programming
Python Descriptors
Python Decorators (class and method based)
Python Buffering Protocol
Python Comprehensions
Python GIL and multiprocessing and multithreading
Python WSGI protocol
Python Context Managers
Python Design Patterns


Advanced topics in python are:

System Programming (pipes, threads, forks etc.)
Graph Theory (pygraph, Networkx etc)
Polynomial manipulation using python
Linguistics (FSM, Turing manchines etc)
Numerical Computations with Python
Creating Musical Scores With Python
Databases with Python
Python Generators and Iterator Protocol
Python Meta-programming
Python Descriptors
Python Decorators (class and method based)
Python Buffering Protocol
Python Comprehensions
Python GIL and multiprocessing and multi-threading
Python WSGI protocol
Python Context Managers
Python Design Patterns

Third party libraries aside here are some:
metaclasses
writing decorators, generators, iterators
writing context managers
C/c++ extensions
Multiprocessing

  • Understand the python object model (at least a passing understanding of metaclasses, slots, and descriptors, as well as how inheritance works), bonus points for recent additions like __prepare__ and __init_subclass__, but also simpler things like when __new__ is useful
  • Understand python's ABCs and inferred types (ie. Iterable, Iterator, Generator, etc.)
  • Understand the c-level data model (ie. at a high level how an int, a list, and a dict are laid out in memory), bonus points if they are actually correct about the way a dict works in cpython, but simply understanding how an unoptimized dict would work is fine.
  • Know why a list comprehension is faster than a for loop (which really is to say understand how bytecode is generated, at high level)
  • Advanced unittesting. Mocks, patches, possibly a more advanced library like pytest
  • Working knowledge of recent features (async/await, type hints)
  • A decent knowledge of the important parts of the standard library: math, itertools, functools, random, collections, logging, sys, os, and threading/multiprocessing/asyncio (I realize these aren't the same, but still). That is, I'd expect a senior dev to know what contextlib.contextmanager, functools.wraps, and itertools.chain were, and when/why one might want to use them. No need to know every function, but where to look at least.
  • A decent knowledge of some non-standard library modules in the domain. This would highly depend on the field, but scipy stack, django/flask/sqla/jinja2, etc.
  • Know at least one sane way to manage environments. This could be a bare venv, or it could be a docker based solution, or a combination, or pipenv, but something
Coroutines (not just generators)
Decorators
Advanced class construction and topics
C/Cython extensions
Data structures
Ability to debug and profile code
Tests


Table of Contents of the book:  Advanced Python 3 Programming Techniques By Mark Summerfield

Section 1: What This Short Cut Covers 3
Section 2: Branching Using Dictionaries 4
Section 3: Generator Expressions and Functions 5
Section 4: Dynamic Code Execution 9
Section 5: Local and Recursive Functions 16
Section 6: Function and Method Decorators 21
Section 7: Function Annotations 25
Section 8: Controlling Attribute Access 27
Section 9: Functors 31
Section 10: Context Managers 33
Section 11: Descriptors 37
Section 12: Class Decorators 42
Section 13: Abstract Base Classes 45
Section 14: Multiple Inheritance 52
Section 15: Metaclasses 54
Section 16: Functional-Style Programming 59
Section 17: Descriptors with Class Decorators 63
Section 18: About the Author 65

http://buildingskills.itmaybeahack.com/book/python-2.6/html/p03/p03c02_adv_class.html
https://python.swaroopch.com/oop.html
www.shahmoradi.org/ECL2017S/lecture/11-python-advanced-decorator-class
 https://www.reddit.com/r/Python/comments/6wl0qk/what_are_the_top_10_key_featuresadvanced_topics/?st=jg5fbksp&sh=b1e49398
http://python-3-patterns-idioms-test.readthedocs.io/en/latest/Metaprogramming.html
https://jakevdp.github.io/blog/2012/12/01/a-primer-on-python-metaclasses/
http://blog.thedigitalcatonline.com/blog/2014/10/14/decorators-and-metaclasses/

SQL security

--create login that uses windows
--authentication and is associated
--with a windows security group
CREATE login [TC\TP_Doctors] FROM windows
--access views to verify that
--the login has been created
SELECT * FROM   sys.server_principals
create a login for a specific windows user
CREATE login [TC\md1] FROM   windows
       --create database users and database roles
       --first activate the database
USE touropharmacy
--find out who is connected now
SELECT Suser_name()
       --set up a user associated with
windows authenticated group login
CREATE USER [TPDoctors] FOR login [TC\TP_Doctors]
       --set up a user associated with
windows authenticated user login
CREATE USER [MD1] FOR login [TC\md1]
       --execute as user = 'MD1'
       --create a server role
USE mastergo
--create server role
CREATE server role [dbOnlyCreator]
--view the types of permissions available on the server level
SELECT *FROM   sys.Fn_builtin_permissions('SERVER')
--view the permissions granted to dbcreator
EXEC Sp_srvrolepermission   @srvrolename = 'dbcreator'
  --assign a server level permission to a login
GRANT CREATE any DATABASE TO dbonlycreator
to view the explicit permissions granted to a server loginSELECT     *
FROM       sys.server_principals PR
INNER JOIN sys.server_permissions PER
ON         PR.principal_id = per.grantee_principal_id
USE touropharmacy
           --create a database role
CREATE role doctorrole
--assign database level permission to doctor role
SELECT * FROM   sys.Fn_builtin_permissions('DATABASE')GRANT
SELECT to doctorrole

       --assign schema level permission to doctor role
DENY SELECT ON SCHEMA::sales TO doctorrole
       --assign table level permission
DENY SELECT ON hr.job TO doctorrole
       --assign object level permission
DENY SELECT ON object::hr.physician(dr_licenseid) TO doctorrole
       --add doctor user as a member of DoctorRole
ALTER role doctorrole ADD member tpdoctors




use [AdventureWorks2014]
GO
DENY SELECT ON [Production].[ScrapReason] ([ModifiedDate]) TO [productionofficer.awuser]
GO
use [AdventureWorks2014]
GO
GRANT SELECT ON [Production].[ScrapReason] ([Name]) TO [productionofficer.awuser]
GO
use [AdventureWorks2014]
GO
DENY SELECT ON [Production].[ScrapReason] ([ScrapReasonID]) TO [productionofficer.awuser]
GO


-- list permissions of all users
SELECT DB_NAME() AS 'DBName'
      ,p.[name] AS 'PrincipalName'
      ,p.[type_desc] AS 'PrincipalType'
  ,dbp.permission_name as 'PermissionName'
      ,p2.[name] AS 'GrantedBy'
      ,dbp.[state_desc]
      ,so.[Name] AS 'ObjectName'
      ,so.[type_desc] AS 'ObjectType'
  FROM [sys].[database_permissions] dbp LEFT JOIN [sys].[objects] so
    ON dbp.[major_id] = so.[object_id] LEFT JOIN [sys].[database_principals] p
    ON dbp.[grantee_principal_id] = p.[principal_id] LEFT JOIN [sys].[database_principals] p2
    ON dbp.[grantor_principal_id] = p2.[principal_id]

WHERE p.type = 'R'

crreate login customerlogin WITH passwrod ="xxx"EXEC sys.Pp_addlogin
  @logname = 'cuistomerlogn'GRANT
CREATE TABLE TO developer_roleDENY
SELECT
ON SCHEMA::humanresourec TO developer_roleGRANT
SELECT
ON prodcution.priciton TO awscusetem_roleDENY
SELECT
ON object::pridction.parent (stanardarcost) TO awcustomerrole exce sp_serverloleperssmion@sernerolname =
'dbcreator'







Thursday, March 15, 2018

Ubuntu play

54.218.49.38

http://forums.fast.ai/t/run-jupyter-notebook-on-system-boot/749/4


http://dlcdnet.asus.com/pub/ASUS/ZenFone/ZX551ML/UL-Z00A-JP-4.21.40.209-user.zip?_ga=2.105212225.318671865.1522430755-1791646875.1522430755

Friday, March 2, 2018

ico






https://github.com/JincorTech/backend-ico-dashboard

https://jincortech.github.io/backend-ico-dashboard/#initiate-password-change-post

docker-compose exec ico systemctl status mongod

https://www.digitalocean.com/community/tutorials/how-to-install-mongodb-on-ubuntu-16-04

https://docs.mongodb.com/tutorials/connect-to-mongodb-shell/


https://docs.docker.com/samples/library/mongo/#connect-to-it-from-an-application

Thursday, March 1, 2018

Backend Developer

Backend Developer with Web development and http, tcp, web-sockets
Modern NoSQL datastores like MongoDB, Redis
Web/mobile application development using Python
Python web framework Worked with MongoDB
Containers or VM like Docker, Kubernetes, Vagrant
Ansible and Jenkins
AWS products: For Compute, Storage, Database or Networking sections
Solr or ElasticSearch
Product startup which have ranked within top 30 in Play Store or iOS App Store



https://blog.codeship.com/using-docker-compose-for-nodejs-development/
https://sub.watchmecode.net/guides/build-node-apps-in-docker/
https://blog.docker.com/2016/07/live-debugging-docker/
https://medium.com/@creynders/debugging-node-apps-in-docker-containers-through-webstorm-ae3f8efe554d

Wednesday, January 31, 2018

Bitcoin Blockchain



Nice Videos:


https://www.youtube.com/watch?v=pLJQy0B5OKo

Tuesday, December 26, 2017

Deep Learning Skills / Data Science

Programming languages (Python, R, Lua, Scala …) and multiple frameworks and technologies (Tensorflow, Torch, Hadoop, Spark, RDBMS…) to support the modeling requirements


Deep learning, other AI, natural language processing, data mining, information theory, and optimization


Python, R, Lua, Scala, C++


Major deep learning libraries:. TensorFlow, Torch, DeepLearning4J


GPU (CUDA), ASIC, or FPGA


Distributed system (e.g. Spark, Hadoop, Ignite …)


Big data visualization

Substantial programming experience with almost all of the following: SAS (STAT, macros, EM), R, H2O, Python, SPARK, SQL, other Hadoop. Exposure to GitHub.
Modeling techniques such as linear regression, logistic regression, survival analysis, GLM, tree models (Random Forests and GBM), cluster analysis, principal components, feature creation, and validation. Strong expertise in regularization techniques (Ridge, Lasso, elastic nets), variable selection techniques, feature creation (transformation, binning, high level categorical reduction, etc.) and validation (hold-outs, CV, bootstrap).
Database systems (Oracle, Hadoop, etc.), ETL/data lineage software (Informatica, Talend, AbInitio)

Data visualization (e.g. R Shiny, Spotfire, Tableau)

AWS ecosystem: experience with S3, EC2, EMR, Lambda, Redshift

Data pipelines  Airflow, Luigi, Talend, or AWS Data Pipeline

APIs:  Google, YouTube, Facebook, Twitter, or Oauth

version control (Github, Stash etc.)


Sunday, December 24, 2017

VBA code for loops play

'Option Explicit

Sub Sample()
    Dim i As Long, j As Long, k As Long, l As Long
    Dim CountComb As Long, lastrow As Long

    Range("G2").Value = Now

    Application.ScreenUpdating = False

    CountComb = 0: lastrow = 6

    For i = 1 To 4: For j = 1 To 4
    For k = 1 To 8: For l = 1 To 12
        Range("G" & lastrow).Value = Range("A" & i).Value & "/" & _
                                     Range("B" & j).Value & "/" & _
                                     Range("C" & k).Value & "/" & _
                                     Range("D" & l).Value
        lastrow = lastrow + 1
        CountComb = CountComb + 1
    Next: Next
    Next: Next

    Range("G1").Value = CountComb
    Range("G3").Value = Now

    Application.ScreenUpdating = True
End Sub



Sub Sample2()
    Dim i As Long, j As Long, k As Long, l As Long
    Dim CountComb As Long, lastrow As Long

    Application.ScreenUpdating = False

    CountComb = 0: lastrow = 6

    For i = 1 To 4: For j = 1 To 4
    For k = 1 To 8: For l = 1 To 12
     
    Cells(i, 20) = i
     Cells(j, 21) = j
      Cells(k, 22) = k
      Cells(l, 23) = l
      lastrow = lastrow + 1
      CountComb = CountComb + 1
     
      Cells(lastrow, 24) = lastrow
      Cells(CountComb, 25) = CountComb
     
    Next: Next
    Next: Next

    Application.ScreenUpdating = True
End Sub

Sub Sample3()
    Dim i As Long, j As Long, k As Long, l As Long
    Dim CountComb As Long, lastrow As Long

    Application.ScreenUpdating = False

    CountComb = 0: lastrow = 6

    For i = 1 To 4
    For j = 1 To 4
    For k = 1 To 8
    For l = 1 To 12
     
    Cells(i, 20) = i
     Cells(j, 21) = j
      Cells(k, 22) = k
      Cells(l, 23) = l
      lastrow = lastrow + 1
      CountComb = CountComb + 1
     
      Cells(lastrow, 24) = lastrow
      Cells(CountComb, 25) = CountComb
     
    Next l
    Next k
    Next j
    Next i

    Application.ScreenUpdating = True
End Sub
Sub playarray()


Dim myThirdColumn As Variant

myThirdColumn = Application.Index(myArray, , 3)



End Sub

' https://usefulgyaan.wordpress.com/2013/06/12/vba-trick-of-the-week-slicing-an-array-without-loop-application-index/

Sub Test()

    Dim varArray()          As Variant
    Dim varTemp()           As Variant
Dim myRng As Range

'Application.Index([A1:E10], , 2) = Application.Index(varArray, , 2)

Set myRng = Worksheets("SheetA").Range("A1:E10")
     varArray = myRng.Value
   varTemp = Application.Index(varArray, , 2)
 '  varTemp = Application.Index(varArray, Array(2, 3), 0)
  '  varTemp = Application.Index(varArray, , Application.Transpose(Array(2)))
   
MsgBox UBound(varTemp) - LBound(varTemp) + 1
    'MsgBox varArray(1, 1)

End Sub


Sub Test2()

    Dim varArray()          As Variant
    Dim varTemp()           As Variant
Dim myRng As Range

'Application.Index([A1:E10], , 2) = Application.Index(varArray, , 2)

Set myRng = Worksheets("SheetA").Range("A1:Z10")
     varArray = myRng.Value
   varTemp = Application.Index(varArray, 3)
    varTemp2 = Application.Index(varArray, , 3)
 '  varTemp = Application.Index(varArray, Array(2, 3), 0)
  '  varTemp = Application.Index(varArray, , Application.Transpose(Array(2)))
   
'MsgBox UBound(varTemp) - LBound(varTemp) + 1
'MsgBox varArray(1, 1)
'MsgBox UBound(varTemp2) - LBound(varTemp2) + 1
MsgBox varTemp2(10, 1)
' VBA Array starts at 1



End Sub



''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Sub Test3()

    Dim varArray()          As Variant
    Dim varTemp()           As Variant
Dim myRng As Range

'Application.Index([A1:E10], , 2) = Application.Index(varArray, , 2)

Set myRng = Worksheets("SheetA").Range("A1:Z10")
     varArray = myRng.Value
   varTemp = Application.Index(varArray, Array(1, 2))
   
    'first two row elements
    'varTemp2 = Application.Index(varArray, , 3)
 '  varTemp = Application.Index(varArray, Array(2, 3), 0)
  '  varTemp = Application.Index(varArray, , Application.Transpose(Array(2)))
   
   
   
'  MsgBox Array(1, 2)(0)
 MsgBox varTemp(1)
 ' the first element actually using array command
 ' the above var temp starts with 1 and not with 0
'MsgBox UBound(varTemp) - LBound(varTemp) + 1
'MsgBox varArray(1, 1)
'MsgBox UBound(varTemp2) - LBound(varTemp2) + 1
'MsgBox varTemp2(10, 1)
' VBA Array starts at 1



End Sub

Thursday, December 14, 2017

Big Data Financial Engineering

Tools and plays



Kafka, Elastic Map Reduce, Avro, Parque, Storm, Hbase


NodejS or Java
- Either:

 Kafka, Storm, Neo4j or Hbase
- Mongoose
- Solr/Lucene

Cassandra, Spark



Deep working experience applying machine learning and statistics to real world problems
Solid understanding of a wide range of data mining / machine learning software packages (e.g., Spark ML, scikit-learn, H2O, Weka, Keras)
Experience with version control systems (git) and comfortable using command-line tools


Preferred:
Knowledge of semantic web technology (e.g., RDF, OWL, SPARQL)
Knowledge of search technologies (e.g., Solr, ElasticSearch)
A link to a portfolio and/or code samples demonstrating your work experience (GitHub, Kaggle, KDD contributions earn major props)



Data Analyst – BI - Training:

Coding data extraction, transformation and loading (ETL) routines.
APIs and databases to pull data together

Hadoop, SQL and NoSQL technologies is required, as well as basic scripting experience in a dynamic language, such as Python or R.
Tools like Jethro, Kyvos, Dremio, AtScale etc.
BI tools like Tableau, Domo, Qlikview etc.
Sata visualization
Relational Databases (eg., Postgres, SQL Server, Oracle, MySQL)
Distributed Databases (eg., Hive, Redshift, Greenplum)
NoSQL Data Frameworks (eg., Spark, Mongo, Cassandra, HBase)
Data Analysis and Transformation (eg., R, Matlab, Python, etc.)

Big Data providers: Cloudera CDH, Hortonworks HDP and Amazon EC2/EMR for deploying and developing large scale solutions.
Hadoop/Spark Big Data Environment Clusters using Foreman, Puppet and Vagrant. Deploy Big Data Platforms (including Hadoop & Spark) to multiple clusters using Cloudera CDH, on both CDH4 and CDH5.
Hadoop MapReduce, YARN, HBase, Spark performance for large-scale data analysis.
Spark performance based on Cloudera and Hortornworks HDP cluster setup in Production Server.
Machine learning data models on Terabytes of data using Spark Ml and Mlib libraries.
 ETL systems using Python, HIVE and Apache spark SQL framework. Storing all the result files in Apache parquet and mapping them to HIVE for Enterprise Datawarehousing.
Real-time data pipelines using Kafka and Python consumers to ingest data through Adobe Real-time Firehorse API into Elastic Search and built real-time dashboards using Kibana.
Aribnb Airflow tool, to run the machine learning scripts in a DAG manner.
Test cases using Python Nose framework.
Scikit learn python scripts to Ml\Mlib spark scripts, which resulted to scalable pipeline framework computing.
PySpark.
Data Pipelines using Spark and Scala on AWS EMR framework and S3.
Real-time Data pipelines using Spark Streaming and Apache Kafka in Python.
Real-time Data pipelines using Apache Storm Java API for processing live streams of data and ingesting to Hbase.
Data pipelines on Cloudera/Hortornworks Hadoop Platform using Apache PIG and automating workflow using Apache Oozie.

Technology: Hadoop Ecosystem /Spring Boot/Microservices/AWS /J2SE/J2EE/Oracle
DBMS/Databases: DB2, My SQL, SQL, PL/SQL
Big Data Ecosystem: HDFS, Map Reduce, Oozie, Hive/Impala, Pig, Sqoop, Zookeeper and Hbase,
Spark, Scala
NOSQL Databases: Mongo DB, Hbase
Version Control Tools: SVN, CVS, VSS, PVCS

Wednesday, September 6, 2017

Advanced Data Science with Python: Machine Learning

Advanced Data Science with Python: Machine Learning

Prerequisites
Knowledge of Python programming and basic features of Python
Able to munge, analyze, and visualize data in Python with Pandas and charting

Syllabus

Unit 1: Introduction and Regression

How to dive into Machine Learning
Simple Linear Regression and Multiple Linear Regression
Forward and Backward Selection
Numpy/Scikit-Learn Lab

Class 2:
Part Classification I
Logistic Regression - Application in Default and other variables
Discriminant Analysis
Naive Bayes
Supervised Learning Lab

Resampling and Model Selection

Cross-Validation
Bootstrap - Breaking it down into simple
Feature Selection
Model Selection and Regularization lab

Class 3:
Classification II
Support Vector Machines SVM
Decision Trees - and Branch Analysis
Bagging and Random Forests
Decision Tree in Python and SVM Lab

Class 4:
Unsupervised Learning - Breaking it down
Principal Component Analysis
Kmeans and Hierarchical Clustering
PCA and Clustering Lab

Recommended Readings

An Introduction to Statistical Learning, by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani
Applied Predictive Modeling, by Max Kuhn and Kjell Johnson
Machine Learning for Hackers, by Drew Conway, John White


R Course Recommended Readings

An Introduction to Statistical Learning with Applications in R, by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani
Applied Predictive Modeling, by Max Kuhn and Kjell Johnson
Data Mining with R, by Luis Torgo
Machine Learning with R, by Brett Lantz


http://www.qcfinance.in/python-for-data-science-machine-learning/

Saturday, September 2, 2017

Django

Django


https://docs.djangoproject.com/en/1.11/intro/install/

https://stackoverflow.com/questions/25716185/page-not-found-404-on-django-site

https://docs.djangoproject.com/en/1.11/howto/windows/

Creating Backend:

https://docs.djangoproject.com/en/1.11/intro/tutorial02/


Need to install SQL

https://www.sqlite.org/download.html


Sunday, August 27, 2017

R Code

#https://bookdown.org/rdpeng/rprogdatascience/managing-data-frames-with-the-dplyr-package.html



library(dplyr)

chicago <- readRDS("chicago.rds")


dim(chicago)


str(chicago)

select(chicago, -(city:dptp))


chicago <- arrange(chicago, date)


head(select(chicago, date, pm25tmean2), 3)

chicago <- rename(chicago, dewpoint = dptp, pm25 = pm25tmean2)

chicago <- mutate(chicago, pm25detrend = pm25 - mean(pm25, na.rm = TRUE))

chicago <- mutate(chicago, year = as.POSIXlt(date)$year + 1900)

years <- group_by(chicago, year)




#https://www.r-bloggers.com/introducing-dplyr/

library(Lahman)
library(dplyr)

players <- group_by(Batting, playerID)
games <- summarise(players, total = sum(G))
head(arrange(games, desc(total)), 5)





#https://www.rdocumentation.org/packages/dplyr/versions/0.7.2/topics/group_by
#for checking group by



#http://www.listendata.com/2016/08/dplyr-tutorial.html

mydata = read.csv("sampledata.csv")


sample_n(mydata,3)

mydata2 = select(mydata, Index, State:Y2008)


mydata7 = filter(mydata, Index == "A")

mydata6 = rename(mydata, Index1=Index)


mydata8 = filter(mydata6, Index1 %in% c("A", "C") & Y2002 >= 1300000 )



summarise(mydata, Y2015_mean = mean(Y2015), Y2015_med=median(Y2015))

dt = mydata %>% select(Index, State) %>% sample_n(10)

t = summarise_at(group_by(mydata, Index), vars(Y2011, Y2012), funs(n(), mean(., na.rm = TRUE)))

t = mydata %>% group_by(Index) %>%
  summarise_at(vars(Y2011:Y2015), funs(n(), mean(., na.rm = TRUE)))



t = summarise_at(group_by(mydata, Index), vars(Y2011, Y2012), funs(n(), mean(., na.rm = TRUE)))

t = mydata %>% group_by(Index) %>%
  summarise_at(vars(Y2011:Y2015), funs(n(), mean(., na.rm = TRUE)))


t = mydata %>% filter(Index %in% c("A", "C","I")) %>% group_by(Index) %>%
  do(head( . , 2))
t = mydata %>% filter(Index %in% c("A", "C","I")) %>% group_by(Index) %>%
  do(head( . , 2))

head(mydata, . , 2)

slice(mydata,3)

t = mydata %>% select(Index, Y2015) %>%
  filter(Index %in% c("A", "C","I")) %>%
  group_by(Index) %>%
  do(arrange(.,desc(Y2015))) %>%  slice(3)

t = mydata %>% select(Index, Y2015) %>%
  filter(Index %in% c("A", "C","I")) %>%
  group_by(Index) %>%
  do(arrange(.,desc(Y2015))) %>%  slice(3)


t = mydata %>% select(Index, Y2015) %>%
  filter(Index %in% c("A", "C","I")) %>%
  group_by(Index) %>%
  filter(min_rank(desc(Y2015)) == 3)


t = mydata %>%
  group_by(Index)%>%
  summarise(Mean_2014 = mean(Y2014, na.rm=TRUE),
            Mean_2015 = mean(Y2015, na.rm=TRUE)) %>%
  arrange(desc(Mean_2015))


#https://cran.r-project.org/web/packages/dplyr/vignettes/programming.html



library(nycflights13)
library(tidyverse)



flights_sml <- select(flights,
                      year:day,
                      ends_with("delay"),
                      distanceis.na(x)is.na(x),
                      air_time )

mutate(flights_sml,
       gain = arr_delay - dep_delay,
       speed = distance / air_time * 60)



arrange(flights, desc(arr_delay))

View(flights)

arrange(flights,dep_delay)


df <- tibble(x = c(5, sin(1), NA))

arrange(df, desc(is.na(x)))

x <- flights %>% mutate(travel_time = ifelse((arr_time - dep_time < 0),
                                        2400+(arr_time - dep_time),
                                        arr_time - dep_time)) %>%
  arrange(travel_time) %>% select(arr_time, dep_time, travel_time)

select(flights,1:5)

arrange(flights, desc(distance)) %>% select(1:5, distance)

flights %>% select(matches("^dep|^arr_time$|delay$"))


x=c(1:4)
y=(1:4)
x==y
x(1,23,4)
x=c('d','d')
x


select(flights,
 air_time,        
       gain = arr_delay - dep_delay ,
         hours = air_time / 60,
          gain_per_hour = gain / hours
)

Friday, July 21, 2017

Web Scraping and Content Mining

DESCRIPTION
Web Scraping and Content Mining
Most interesting course in NYC.
2 sessions workshop
Web Scraping is a method for extracting textual characters from websites so that they could be analyzed. Web scraping is sort of content mining, which means that you collect useful information from websites, including quotes, prices, news company info, etc.This method for gathering data is direct, either through looking at websites' html code or visual abstraction techniques using Python programming language.
We start workshop by exploring different methods to gather data from Web. We go through the whole process of gathering, storing and analyzing data. For our examples we use real-life financial quotes and Annual reports 10-K. During the course we learn how to use numerous Python libraries - Urllib, Requests, Wget, BeautifulSoup 4.0, SSL, PDFminer3k, Twitter and others.
Also, we learn to constract Regular expressions patterns to find targeted information on Web pages. As a part of content mining, we build Twitter application to search and analyze the trends.
The price is for two classes:
You will Learn:
BeautifulSoup Python Library
How to use Urllib and Requests
Regular Expressions patterns
Read and analyze PDF files
Store Data with CSV files and SQL Database
Create Twitter app
Build Custom Google Search Engine

Sunday, July 16, 2017

Craiglist Adds

Financial Modeling Tutor $25/hr (Midtown)



I am a former investment fund analyst and experienced in investment banking. 


Offering lessongs on excel, especially build models for companies.

Very affordable rate of $25, and you will be given all the skills needed to land a job at a hedge fund, investment bank, or private equity firm.

Valuation Methods for Companies, putting together models, write-ups, and presentations.

This is a very limited service and is temporarily offered for this month while I am vacation and interested to share what I learned.

Take advantage while you can, hours are limited, availability also at 6 pm - just after your office.

Feel free to contact me if you are interested in learning how to be an finance/excel expert! Thank you!    


Google Sheets and App Script (JavaScript) tutor $25/hr (Manhattan)

Hi,
I am a tutor for Google App script (Java script) used for automation in Google sheets. If your company is using google sheet learning automation will help you to progress. Also, important for business students.
Charge $25/hour.
Location: All in NYC area.
Please get in touch

.    

GRE / GMAT Quant / Quantitative/ Math Tutor $25/hr

GRE / GMAT Quant Tutor
Experience in tutoring, videos online and good references.
Price $25 per hour
5 years of exp in quantitative GRE GMAT tutoring
From the author of FreeGREGMATClass dot com
Check out the youtube videos contributed by students and tutors for FGGC
First session of 30 minutes is free.   



Quantitative Math Excel VBA $25/hr (New York)

Tutoring Data Science has been my hobby and recreational activity. Many tutoring projects are volunteering and networking oriented (getting new insight). I do it so that I can revise what I do at school and at work. I am an Electrical Engineering graduate, GAARP-FRM certified, PG Dip in Fin analysis and Risk Management, cleared CFA L1, International - MBA (15-16).

I have extensive teaching experience with students of various profiles and backgrounds where I have learned and enhanced my skills and also helped learners. I am also an extremely friendly person. I have taken many online classes as tutor on qcfinance.in

QcFinance.in has tutors available for meetups (mix of onsite at home & online through Wiziq/Skype/Adobe-connect & custom videos support emailed to you for your doubts on holidays)

Subject areas teach includes: Quantitative Methods for GRE / SAT / GMAT, CFA L1, FRM L1, MATLAB/R/SPSS for Quant, Excel-VBA Programming. Quantitiative and Analytics Excel programming & VBA programming.

Sub topics:

Basics of Excel: Vlookups, Hlookups, Index Match, Dependents, Data tables (1 way and 2 ways), Pivots, Charting, Filters, SQL integration, VBA coding, Address and indirect, Offset, Array functions, etc.

Applications: Regression, Histograms, Monte carlo simulation, rank correlation, dashboards, more.
Automation using VBA: Loops (for, do while, case), Recorder, Arrays and Matrices, if else, indexing, etc.

Website: www.qcfinance.in

Playlist of sample quant videos: https://www.youtube.com/playlist?list=PL_-KSXJS5pxOiLjAoe5uAHPAsv-UhIM7i

Keywords: Quant Trainer, Tutor, Trainer, Teacher, Home tuition, GRE, GMAT, Quant, Programming, CFA Level 1, FRM Part 1, Mathematics Tutor, Maths, Onsite.
Keywords: Trainer, Tutor, Home tuition, Excel, VBA, Onsite, trainer, help assignment, video solutions, In Person, 1 on 1, Home Tutor. 

Office Automation on Excel VBA Python SQL R (new york)

We provide official automation services on Excel VBA Python SQL R. 

Automation is the next biggest revolution, legacy methods if not replaced by automation will reduce the productivity of the firm which might even lead to extinction.

Our can reduce a lot of manual work and use lot of Excel Analytics features, our clients have reduced work by upto 50-70% which helped me focus on their product and other value addition to their core business.

Get more hours from your employees and more robust analytical framework!

Please contact me for more details about various processes that we can automate.    


Statistics, Data Science, Machine Learning, Statistical Computing, R   


Tutoring for statistics, machine learning, and data science. The focus includes statistical theory as well as its application, building models. That includes the following: 
•Theory Courses: Probability, Statistical Inference, Bayesian Statistics, Decision Theory, Point Estimation, High-Dimensional Inference, Time Series, and other MS/Ph.D. courses

•Machine Learning: Ridge Regression, LASSO, Basis Pursuit, Supervised/Unsupervised Learning, Neural Network, Statistical Learning Theory 

•Social Science: Causal Inference, Hierarchical Model, Multiple Imputation, Matching

•Statistical computing: R programming, Matlab, STATA, Python, Java, C